Ben Nolan2017-07-24T00:37:47+00:00http://bnolan.github.com/Ben Nolanben@nolanconsul.comUsing SSE to make things live2017-07-24T00:00:00+00:00http://bnolan.github.com/2017/07/24/using-sse-to-make-things-live<p>Last week I stayed up late on a Tuesday night and built the prototype of <a href="https://talking.herokuapp.com">Talking</a>, my idea of a forum where you leave voice messages for each other. I decided I wanted to do this as a way to get out all my ideas for a next generation reddit out on one place:</p>
<ul>
<li>Live updates</li>
<li>Voice replies</li>
</ul>
<p>That’s basically it, I wanted to do those two things, in preact. Anyway, I’ve got it mostly working, you can hear me talking to myself (since no one else uses it), I haven’t implemented voting yet, but I will do that soon, and then you’ll be able to use it the way I imagine (I’ll use the <a href="http://www.evanmiller.org/how-not-to-sort-by-average-rating.html">Lower bound of Wilson score confidence interval for a Bernoulli parameter</a>), as a shitty weird forum that no one uses. Or maybe I’ll post it on /r/internetisbeautiful and 10,000 people will use it for one day and then never come back. Or maybe it’ll develop some kind of weird synthesisers user group who use it to reply to each other with beeps and boops (I know I’d like to use it like that).</p>
<p>Anyway - so I’ve got it going, and how it works, is the <a href="https://github.com/developit/preact">preact</a> app boots up and fetches the recordings, then you can click a recording to get the tree or replies.</p>
<h2 id="getting-a-tree-of-recordings">Getting a tree of recordings</h2>
<p>I do this in one operation using this recursive query:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>WITH RECURSIVE recordingtree AS (
SELECT r1.*, 0 as depth
FROM recordings r1
WHERE r1.id = $1::integer
UNION ALL
SELECT r1.*, r2.depth + 1 as depth
FROM recordings r1
INNER JOIN recordingtree r2
ON r2.id = r1.parent_id
)
SELECT * FROM recordingtree;
</code></pre>
</div>
<p>That returns a post and all it’s children (using <code class="highlighter-rouge">parent_id</code>), and then because I couldn’t work out how to munge that into json in postgres, I do a little postprocessing in node land to turn it into a nested json object.</p>
<p>All well and good, but - how about live updates?</p>
<h2 id="live-updates">Live updates</h2>
<p>My dream for a better reddit (or one that’s impossible to follow, or really hard to keep up because it uses too much resources, but anyway, my idea) is to see people replying directly, like see ‘x is recording a reply’ and then see the reply appear instantly. My idea here is genius:</p>
<h2 id="whenever-you-return-json-from-a-fetch-make-it-a-sse-stream">Whenever you return json from a fetch, make it a SSE stream</h2>
<p><a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events">Server sent events</a> are basically the thing you should always be using instead of websockets. Websockets are cool, but totally unnecessary 95% of the time (unless you’re making scenevr, that needed websockets for realtime position updates from client to server).</p>
<p>So what I’m planning on doing, is when a client requests some data (like the list of top level recordings, or all the recordings in a tree), is to do the query, send the response, and then keep the request open (it’s really easy to do SSE with express), and then subscribe to a redis pub/sub channel with the name of the request (eg <code class="highlighter-rouge">/recording/1234</code>), and then whenever someone submits a new recording (or votes or whatever), you:</p>
<ul>
<li>Write the new recording to the database</li>
<li>Query for the tree of recordings</li>
<li>Convert the tree to json</li>
<li>Write the json to a redis pubsub topic</li>
</ul>
<p>All connected clients are listening to that topic, and we can just send the json down the wire (plus two newlines <code class="highlighter-rouge">\n\n</code>) and they’ll all get a live updated view of the thread.</p>
<p>It’s like the easiest way ever to do real time live stuff (the same way github does). Anyway, I’m gonna try to that this week, we’ll see how it goes.</p>
Bonkers - microblogging with html2017-06-08T00:00:00+00:00http://bnolan.github.com/2017/06/08/bonkers---microblogging-with-html<p>So I couldn’t sleep last night (thanks jet lag), so I sat in bed and prototyped BONKERS, a microblog that lets you paste full html.</p>
<p><img src="http://i.imgur.com/TntXtOH.png" />
<cite>Using font sizes and emoji</cite></p>
<p>I didn’t do any server backend for it, so this is all just client side stuff. The interesting part is how you render the tweets, or toots, or posts, or whatever you call them.</p>
<p>What I settled on was using an <code class="highlighter-rouge"><iframe /></code> with a <code class="highlighter-rouge">datauri</code> src attribute. This embeds the content on a null uri. I also looked into using the <code class="highlighter-rouge">sandbox</code> attribute, but that doesn’t work 100% on safari (you can still run javascript that opens modals). So, what I think I’m going to do - is by default, content displays in an iframe, with the code having been sanitized (so that there is no javascript in it). However, you can still use full css and css animations, embed images etc. However, once you click on a post, I might reload the iframe with javascript enabled, so that you can have little embedded games, but it prevents autoplaying that might allow people to be a pain in the ass.</p>
<p>I have a little api exposed to the iframes using <code class="highlighter-rouge">window.postMessage</code>. At the moment there is only a <code class="highlighter-rouge">resize</code> message type (so that the iframe resizes to the size of your post), but in the future, i’d like to have the ability to create posts or follow people through this api (and then if I get the security wrong, people can write sweet worms that autofollow each other and take down the site woo!).</p>
<p>Another thing I’m a bit unsure about, is whether to force same origin for images and other media. I don’t really know if it’s possible (almost definitely not possible when javascript is enabled, but probably is through a sanitizer), and force people to upload content to the microblogging service. This would make it easier to prevent linking to really offensive images, but it also sort of breaks the web, maybe it’s better just to let people use any content from wherever.</p>
<p>I also added a ‘fork’ button. All the content on BONKERS would be creative commons licensed (maybe you can pay 10 cents per post if you want to remove the CC license), so anyone would be able to remix your post. In terms of editors, I think a simple WYSIWYG editor would be good (bold text, add links, change colors and fonts), as well as a markdown editor (for noobs) and a full html editor with syntax highlighting (for l33t haxx0rs).</p>
<p>Anyway, was fun prototyping.</p>
Twitter with Markdown2017-06-07T00:00:00+00:00http://bnolan.github.com/2017/06/07/twitter-with-markdown<p>This is an idea that keeps percolating up from the back of my mind. I want to build something like twitter (public feeds, follow people, microblogging), except it supports full html. So you can tweet something and use <strong>fonts</strong> and <strong>colors</strong> and have links that do stuff. Obviously you’d need to sanitise the posts (and probably load them in an iframe as a datauri with no permissions), but I think it’d be fun to be able to post interactive content. You could post a little survey, or a game (maybe you could use something like amp and have a static view of the post, but when you click on it, it can open a sub-iframe and run some javascript). It’d be little tiny bits of the web, streamed to you in realtime.</p>
<p>It’d be even better if you had some kind of API so that the “tweets” could repost themselves (with your permission). For example, you might play an embedded game of tic/tac/toe, win the game, then post a message saying ‘yay I won! you have a go!’.</p>
<p>Also, it’d be great if you made all the content creative-commons by default, so that there was a ‘remix’ button on peoples tweets, so you could post a cool little design, and someone could say ‘hey that would look great in monochrome with roboto’, they could edit your post and then repost. It’d be like github.</p>
<p>I shall call it…</p>
<p>git-twitter-html-microblog</p>
<p>Hmmm… I wonder if that .com address is available.</p>
Porting CPP2017-05-19T00:00:00+00:00http://bnolan.github.com/2017/05/19/porting-cpp<p>So ever since I <a href="//scenevr.com">shut down</a> SceneVR, I’ve been free to work on a variety of different ideas. One of the things that was always floating around in the back of my head, was writing something like <a href="//janusvr.com">Janusvr</a>, except designed to run on mobile VR instead of desktop.</p>
<p>That’s why I started work on Fontus, by c++ renderer. It’s written in c++ so that it’s:</p>
<ul>
<li>Fast</li>
<li>Extensible (can use any crossplatform c / c++ libraries)</li>
<li>Native (so it can be listed in the oculus / steam / play store)</li>
</ul>
<p>And although making c++ compile on multiple platforms is a huge pain in the ass (I started on mac, porting to windows wasn’t too hard, but porting to android with the NDK has been a real struggle), I’m enjoying the environment once I have things up and running.</p>
Clang and Android2017-05-18T00:00:00+00:00http://bnolan.github.com/2017/05/18/clang-and-android<p>As my side project, I’ve been working on a <a href="http://fontus.bennolan.com/">c++ renderer</a> for <a href="http://fontus.bennolan.com/a-minus.html">a-minus scenes</a>. This has been a super fun project, using c++11, three.cpp and opengl to make a pretty lightweight and nice and speedy renderer.</p>
<p>I’ve written renderers for xml markup before using unity, but I don’t really enjoy using unity that much, because it’s quite far away from my ideal toolchain (git, command line and sublime text). Using c++11 and cmake, I’ve been enjoying developing on the command line. I still have to dig out an IDE from time to time (like using xcode to get proper stack traces), but mostly it’s been all command line (so in theory I can automate the builds on CI). A lot of unity debugging is ‘do all this in the gui and it’ll work again’, I much prefer the ‘search for error, find answer on stack overflow, copy paste’ style development that I get with C++.</p>
<p>Also, c++ is super fast, and it’s really easy to merge in helper c libraries (like the tiny html renderer, <a href="https://github.com/litehtml/litehtml">litehtml</a> that i use).</p>
<p>Anyway, last night and today, I’ve been trying to get my project to build for android. After a few false starts last night, I realised that I can’t get it to build using a <code class="highlighter-rouge">Cmake</code> (sdl only has <code class="highlighter-rouge">ndk-build</code> support), but after fighting that losing battle for a while (I really wanted to be able to use one set of cmake files for all platforms, but cest la vie), I’ve finally got the app <em>starting</em> to compile - even if the clang version for ndk refuses to enable c++11 support:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>error: no type named 'unique_ptr' in namespace 'std'
std::unique_ptr<std::stringstream> stream;
</code></pre>
</div>
<p>I feel like this is going to be a long battle, but I’ll keep at it, and hopefully I’ll have a gear vr build of Fontus in a week or so.</p>
It's on2017-05-17T00:00:00+00:00http://bnolan.github.com/2017/05/17/it's-on<p>So it looks like this blog still works. Maybe I’ll start blogging again.</p>
Firing up the blog2017-05-17T00:00:00+00:00http://bnolan.github.com/2017/05/17/firing-up-the-blog<p>Is this thing still on?</p>
<p>I just found a folder in my homedirectory called <code class="highlighter-rouge">Blog</code>, and a command in my path called <code class="highlighter-rouge">newpost</code>, so I’m just seeing if this is all hooked up.</p>
<p>Writing blog posts in markdown with sublime text is dope! No need for <a href="//medium.com">medium</a>.</p>
SceneVR, multiuser 3D2014-12-10T00:00:00+00:00http://bnolan.github.com/2014/12/10/scenevr-multiuser-3d<p><a href="//scenevr.com">SceneVR</a> is my node.js + webgl project that lets you easily create 3d scenes that you can experience with multiple people at once. It uses node.js and websockets to provide a server. The server loads <a href="http://www.scenevr.com/scene.html">html-like</a> scene descriptions and lets you script interactions, so that the scene can respond to people in the scene. The cool thing is, that everything is simulated on the server, so every connected client sees the same thing. This makes it easy to write multi-user experiences (games, demos, toys), just code it once in xml + javascript, and everyone can connect to it and play with it.</p>
<p>It’s an open source (bsd licensed) project and is up on <a href="//github.com/bnolan/scenevr">github</a>. I don’t have a live demo of it yet, because I’m still nailing down a few things before I stick it up on heroku or nodejitsu. Because the hosting is a bit different from normal node.js apps, I’d like to make a hosting service that specializes in serving sceneVR scenes.</p>
<p>The <code class="highlighter-rouge">virtualreality</code> folder on my macbook has had lots of activity over the past 6 months, but it was only during a holiday to visit <a href="//twitter.com/ristari">Rissa</a>s family in Brisbane in early November that I decided to have a crack at building a high quality scene server in node.js, and knock up a quick web client to view the scenes. It came together pretty fast, using <a href="https://www.npmjs.com/package/dom-lite">dom-lite</a>, htmlparser2 and websockets.</p>
<h2 id="current-state">Current state</h2>
<p>Currently SceneVR is pre-release. I’ve got a milestone set up in github, and once I’ve closed all those issues, I should be good to start showing it off to more people. I really want to make a good video that demos everything you can do in SceneVR, since there’s a lot to demo (realtime reload, index files, javascript, multiple clients, physics models, model import, lightmaps, audio).</p>
<h2 id="oculus-rift-support">Oculus rift support</h2>
<p>The rift is supported in webVR browsers (chromium and firefox nightly). It’s not 100% yet, but works pretty well. I test things in webVR regularly, and it helps me remember to make sure everything is rendered at a reasonable size, that the player moves at a reasonable speed, and that the html-textures aren’t too small to be read on a dk1 screen</p>
<h2 id="mobile-support">Mobile support</h2>
<p>This was a bit of a surprise to me, but the client renders perfectly on mobile. I added two little thumb wheels for controlling the look and movement of the client. It’s not finished either, but I definitely want to keep mobile as a platform for viewing sceneVR scenes.</p>
<h2 id="portals">Portals</h2>
<p>SceneVr has a <code class="highlighter-rouge"><link /></code> element that links between scenes. So you can walk around a scene, find the link, a portal opens to the next scene, you step through the portal and you’re in the next scene. I got this idea from <a href="//janusvr.com">JanusVR</a> which does something similair, and of course the great game “Portal”.</p>
<h2 id="familiair-api">Familiair API</h2>
<p>I’ve modelled the scripting API for scenes on the DOM. So you can <code class="highlighter-rouge">document.getElementById</code> and <code class="highlighter-rouge">element.addEventListener('click', ...)</code>. You can also manipulate the <code class="highlighter-rouge">position</code>, <code class="highlighter-rouge">rotation</code> and <code class="highlighter-rouge">scale</code> vectors directly (well rotation is a euler, but yeah…). So <code class="highlighter-rouge">element.position.z += 10</code> just works.</p>
<h2 id="network-updates">Network updates</h2>
<p>Network updates work magically, and send updates 5x a second (the client sends the players location that often too). The wire protocol is currently xml, which isn’t ideal, but I plan on optimising the most frequent messages in time. I use the tween.js library to interpolate scene updates so that everything appears silky smooth (at 60fps even on my underpowered macbook 11”).</p>
<h1 id="future">Future</h1>
<p>I plan to keep working on SceneVR, and will probably start a blog at <a href="//scenevr.com">scenevr.com</a> to post updates there. I’d love to eventually work part-time on SceneVR if enough people start creating content for it, but that’s a long way off at the moment. At the moment the project has only 10 stars on github. Here’s to the next 10! :)</p>
Herenow feature requests2014-09-25T00:00:00+00:00http://bnolan.github.com/2014/09/25/herenow-feature-requests<p>I’ve been given a bunch of feedback on Herenow. I thought I’d make a blog post to list the main feature requests that my friends and other users have given me.</p>
<h1 id="reply-to-message">Reply to Message</h1>
<p>People want to be able to click on a message and see a bit more detail about it, and be able to reply to the message, which will send a notification to the person who wrote the message, even if that person is no longer at the location where they left the message. So if someone leaves a message that says “the honest rob burger is amazing” at <a href="https://www.facebook.com/pages/Ekim-Burgers/201864206519644">Ekims</a>, you can reply to that person and say ‘awww damn thanks for the recommendation the blue cheese aoili was amazing yo’.</p>
<h1 id="less-battery-usage-on-ios">Less battery usage on iOS</h1>
<p>I think I fixed this in the 1.2 release, but some friends are still reporting that Herenow is just constantly using the GPS on their phone. I’ll need to look more into this. I’d also like to do that trick where you poll the accelerometer to see if you are stationary before hitting up the GPS, I don’t know if this would work for Herenow though.</p>
<h1 id="open-messages-from-notifications">Open messages from notifications</h1>
<p>If you get a notification while you’re driving, when you click on the notification, it doesn’t show you the relevant message, it shows you the messages from wherever you are now, which is annoying, you never get to collect those messages.</p>
<h1 id="ping-people-nearby">Ping people nearby</h1>
<p>There should be a button where you can ping to see if any other herenow users are nearby, without actually having to write a message. Then you could start a conversation with them.</p>
<h1 id="allow-shadowbanning">Allow shadowbanning</h1>
<p>I currently don’t have any UI for banning offensive users. This needs to happen. Basically, if 2-3 people flag a message, that message should show up in some UI on my phone, and I should be able to shadowban the user so they’re not offending other people.</p>
<h1 id="allow-purchasing-a-username">Allow purchasing a username</h1>
<p>So that businesses, or people with a spare $2 can purchase a username, and not post anonymously, but as <code class="highlighter-rouge">captainbenis</code>, or whatever.</p>
<h1 id="yik-yak">Yik Yak</h1>
<p>I know you’re not meant to compare your app to anyone elses, you’re only meant to focus on your users and your todolist, but I keep checking in on yik yak from time to time. It’s massively popular in US universities, dozens of messages a minute. I think herenow is quite different from Yik Yak, but Yik Yak is basically exactly what I had in mind when I posted this <a href="https://twitter.com/bnolan/status/336354335388749824">tweet</a> back in May 2013, except I found Spraffl, and decided not to try and build it myself.</p>
<p>Maybe that’s a good reason to build and push herenow properly, so that the next company to come along and build a good leaving-notes-at-location app is my company, not some other startup. Hah. :)</p>
Working collision map on mapbox2014-08-27T00:00:00+00:00http://bnolan.github.com/2014/08/27/working-collision-map-on-mapbox<p>So this is a total hack, but it works. <a href="/experiments/walk-te-aro/">Try it out</a>. Basically, I’ve exported the json that represents the static polygons in the physics simulation, and hardcoded the translation functions to convert physics-space to world space. I also hacked up box2d to increase the maximum translation speed (I probably need to scale everything down to do it properly). I also made the streets too wide in the physics. I might go back and fix that. But yeah, you can get a feel for how the game engine might work, with being able to walk up and down streets and side streets. I think it could be a pretty cool game mechanic.</p>
Physics simulation of Wellington2014-08-25T00:00:00+00:00http://bnolan.github.com/2014/08/25/physics-simulation-of-wellington<p>So I did a bit of work on <a href="http://tuesdayapp.com">Tuesday</a> today, but also did a good afternoons work on my virtual tourist game. <a href="/experiments/wellington/">Here is the latest</a>.</p>
<p>It takes ages to load (about 40 seconds) because the code that combines the streets is horrendously slow (23 seconds). I think I can optimize it, I suspect I’m just using jsclipper incorrectly (feedback sought and appreciated!). Once the page has finished loading, you can walk around the streets of Wellington. I’m not sure why the speed of the little red arrow is so slow, box2d seems to ignore the strength of the impulse (I guess it’s normalized at some stage?). Anyway. The goal is to combine this physics model with the <a href="/experiments/walk/">walking demo</a> to make a cool webgl powered walking simulator of a city, where you can visit locations to get points, or collect “memories”.</p>
<p>If you’re trying to make the code work on your own, note that I had to have hacked up <code class="highlighter-rouge">createBucket</code> to always set <code class="highlighter-rouge">bucket.interactive = true</code> - since that didn’t seem to work correctly in the current master branch on mapbox-gl-js. This is taking way longer than I expected - I hope this is a cool game mechanic walking around the city. :)</p>
Walking physics2014-08-23T00:00:00+00:00http://bnolan.github.com/2014/08/23/walking-physics<p>I’ve done some more work on trying to make a walking simulator on openstreetmap data. I used jsclipper to draw a big rectangle, then extruded the streetlines and cut them out of the rectangle, then turned the polygons into convex hulls and loaded them into box2d. <a href="/experiments/physics/">See here</a> for a demonstartion. The code is super ugly at the moment.</p>
Mapbox GL walking game2014-08-22T00:00:00+00:00http://bnolan.github.com/2014/08/22/mapbox-gl-walking-game<p>I discovered <a href="https://github.com/mapbox/mapbox-gl-js">mapbox gl</a> for the web the other day. It’s an awesome library by mapbox that renders vector tiles in the browser using webgl. It’s super fast, and allows rotation of the map, which let me prototype something I’ve been trying to do for ages.</p>
<h1 id="walking-simulator">Walking simulator</h1>
<p>I want to make a tourism game, where you walk around a city, collecting photos of places, buying beers at pubs and eating lunch at cafes. It’s like an RPG, except it’s set in real cities, and you explore by walking the streets. I was thinking something like Game Never Ending. You can use foursquare to populate the places database, panoramio and google street view to show photos of the places (walk past a cool church in Prague, collect a photo and get some in-game points).</p>
<h1 id="prototype">Prototype</h1>
<p>This <a href="/experiments/walk/">prototype</a> shows what I’m thinking in turn of UI, use the arrow keys to walk and turn.</p>
<h1 id="physics">Physics</h1>
<p>I haven’t worked out how yet, but in theory, it should be possible to get the vector street data out of the mapbox tiles. I’ll then use the jsclipper library to draw a big rectangle over the city and cut out all the streets. Once I’ve converted the left over polygons into convex polygons, I can use matter.js or box2d to simulate physics (with gravity set to 0), then you should be able to push in any direction and the physics engine will ensure you stay on the streets, without having to aim exactly down the street.</p>
<p>I got jsclipper working generating the concave polygons, but didn’t have enough time last night to convert the polys to convex and throw them into box2d. That’ll be the task for the weekend.</p>
<h1 id="gameplay">Gameplay</h1>
<p>I’m quite excited about this game, since it lets you simulate trips to cities that you might never be able to visit (a real problem down here in New Zealand, where it’s at least 30 hours to Europe). I’ll probably make the game run at 1 second = 1 minute, has in-game currency for buying things. I’d also like to make it multiplayer, so that if someone else is exploring Wellington at the same time as you, you’ll see them walk past you. Anyway. I’ve had these ideas for a while, it’s exciting that mapbox GL let’s me prototype them.</p>
Deploying node.js apps2014-08-11T00:00:00+00:00http://bnolan.github.com/2014/08/11/deploying-node.js-apps<p>I’ve got a few node.js apps kicking around on my computer that I’ve been wanting to deploy for a while, but I’d never got around to it. Happily, I decided to try and deploy the <a href="https://github.com/bnolan/mv-asset-server">asset-server</a>.</p>
<h1 id="forever">Forever</h1>
<p><a href="https://github.com/nodejitsu/forever">Forever</a> is a little tool by nodejitsu that starts node.js apps and makes sure they keep running, even if they crash. It’s pretty straightforward and seems to work reliably. It had a bunch of parameters for logging and restarting the server. I tried it out and it seems pretty good.</p>
<h1 id="git-archive">Git archive</h1>
<p>I was looking for <a href="http://twitter.com/sminnee">sam minnee</a> article on deploying using git, but couldn’t find it, so instead I went for the easy answer:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git archive | gzip > bundle.tar.gz
</code></pre>
</div>
<p>AFAICT, this takes the current master branch and archives it up.</p>
<h1 id="bash-magic">Bash magic!</h1>
<p>So then I just needed a little bash scripts to create the bundle, <code class="highlighter-rouge">scp</code> it to my production server, untar it to <code class="highlighter-rouge">/app/$ISODATE</code>, symlink it in place, run <code class="highlighter-rouge">npm install</code>, then start and stop <code class="highlighter-rouge">forever</code>. This is the script I used:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
git archive master | gzip > /tmp/asset-server.tar.gz
scp /tmp/asset-server.tar.gz munich:~/builds/
<span class="nv">RELEASE</span><span class="o">=</span><span class="sb">`</span>date -u +<span class="s2">"%Y-%m-%dT%H:%M:%SZ"</span><span class="sb">`</span>
ssh munich <span class="sh"><<ENDS
cd ~/apps/asset-server/current
forever stop asset-server.js
mkdir -p ~/apps/asset-server/$RELEASE
cd ~/apps/asset-server/$RELEASE
tar xvfz ~/builds/asset-server.tar.gz
rm ~/apps/asset-server/current
ln -s ~/apps/asset-server/$RELEASE ~/apps/asset-server/current
mkdir -p ~/apps/asset-server/models
mkdir -p ~/apps/asset-server/current/public
ln -s ~/apps/asset-server/models ~/apps/asset-server/current/public/models
cd ~/apps/asset-server/current
npm install
forever start -a -l asset-server.log asset-server.js
ENDS
</span><span class="nb">echo</span> <span class="s2">" * Deploy complete"</span>
</code></pre>
</div>
<p>It seems to work for now, it’s not ideal, but it’ll do.</p>
<p>nb: An earlier version of this post had the <code class="highlighter-rouge">RELEASE=...</code> line inside the heredoc, which didn’t do what I expected. The above code is corrected.</p>
AOC 4k Monitor2014-08-08T00:00:00+00:00http://bnolan.github.com/2014/08/08/aoc-4k-monitor<p>I bought one of the new AOC 4K Monitors (a u2868pqu). It’s a great monitor, and available for only $800 in New Zealand. I ordered mine from mightyape and it arrived the next day.</p>
<h1 id="macbook-air-11">Macbook air 11”</h1>
<p>I tired an equivalent monitor on my 11” air and it wouldn’t even power up, so no go with the HD3000.</p>
<h1 id="macbook-retina-15">Macbook retina 15”</h1>
<p>I have a 15” retina with the 650m GPU, and I plugged the monitor into the HDMI port and it just worked right away. You even have the option of using ‘more space’ mode, with is 3000px across (equivalent) with retina pixel subsampling. It looks great, and has plenty of space. The only problem is that it runs at 30hz, not 60hz. The same problem, 30hz, when plugged into the displayport port on my macbook too.</p>
<h1 id="macbook-retina-13">Macbook retina 13”</h1>
<p>I tried out a new (2014) 13” retina, plugging the monitor into the mini display port, and still the asme problem, only 30hz. I’m not sure if this means the cable we’re using doesn’t support DP1.2, or the monitor just won’t do 60hz plugged into a mac.</p>
<h1 id="gaming-pc">Gaming PC</h1>
<p>I ordered a gaming pc (from <a href="http://www.pbtech.co.nz/index.php?z=p&p=WKSPB2107&name=PB-2107-Upgrade-box-Intel-New-Haswell-Core-i5-4590">pbtech</a>), I ordered it with a GTX 750 ti, which has displayport 1.2 and should have enough power to easily drive normal operating system stuff at 60hz. I’ll find out next week when the pc arrives. I’ve also ordered windows 8.1 since it has hidpi mode (as does sublime text and visual studio 2014). I doubt i’ll be able to do any gaming at 4k, although I will try really old games like quake 2. Maybe even TF2 if I run it at super low settings.</p>
Oculus SDK and Three.js2014-08-06T00:00:00+00:00http://bnolan.github.com/2014/08/06/oculus-sdk-and-three.js<p>At work today, I did a demo of some of the VR stuff I’ve been hacking on. It was pretty cool, everyone was way more interested in the 3D / VR stuff than they were in my other projects, which made me think that I need to get my shit together and release some of this stuff.</p>
<h1 id="timewarp-and-threejs">Timewarp and three.js</h1>
<p>One of the things that’s been on my mind, is that even if I build a cool multiuser 3d demo, I still have to make it work with the oculus rift somehow before I can post it on the <a href="https://developer.oculusvr.com/forums">oculus forums</a>. I’ve already done a prototype using <a href="https://github.com/benvanik/vr.js/">vr.js</a> which uses the NPAPI (deprecated but still works) to talk to the rift SDK and expose positional info to javascript. It works well, but it’s pretty laggy (compared to the tuscany demo which uses unity). I’ve been wondering if there’s anyway to use timewarp (where the scene is rendered to a depth buffer, then the oculus is polled to get the most recent location and the buffer is warped to match your new head location, reducing latency) with three.js. The only way I can think of at the moment would be to somehow hack up a chromium build that passed the webGL output to the SDK for final warping and sending to the rift.</p>
<h1 id="node-webkit">Node-webkit</h1>
<p>A cool dude called loktar created a demo of using <a href="https://developer.oculusvr.com/forums/viewtopic.php?f=39&t=4727">node-webkit with vr.js</a> to create a packaged file that you can download and try out virtual reality in javascript. It’s a cool idea, but I couldn’t get it to work in windows 8.</p>
<h1 id="webvr">WebVR</h1>
<p>And this other dude <a href="http://blog.tojicode.com/2014/07/bringing-vr-to-chrome.html">Brandon Jones</a> has been working on adding webVR support to chrome. He’s already got chrome builds that support the webVR standard. Firefox is also working on webVR. In Brandons post he notices that the latency of using webVR in chrome is 64ms (4 frames) from head movement to updating the display, whereas the Tuscany demo on the same box has a 48ms latency. There’s no discussion of getting timewarping to work in the browser, but there will be work about speeding up the rendering pipeline</p>
<h1 id="conclusion">Conclusion</h1>
<p>The conclusion is that there is cool work being done on making the browser a viable platform for doing stuff with virtual reality headsets, and I need to get my act together and opensource and deploy some of the code I’ve got kicking around.</p>
Physics, hulls and cannon.js2014-07-24T00:00:00+00:00http://bnolan.github.com/2014/07/24/physics,-hulls-and-cannon.js<p>As part of my exploring metaverse stuff lately, I’ve been thinking of ways I could make some cash moneys off the technology I’ve built. One of the first ideas I’ve come across, is to make a walk-through visualizer for architectural models. So as well as providing a global metaverse where anyone can upload art, also have private workspaces where people can place models and walk through them with their clients.</p>
<h1 id="collision-detection">Collision detection</h1>
<p>This is where the problem is. I’m pretty sure I can get cycles light-baking working (the hardest part imo is automatic UV mapping), so I can do a lightmap for the 3d model (it’s amazing how good a cheap sketchup model looks when you put it through a proper GI renderer - that was one of my takeaway lessons from working at <a href="http://indigorenderer.com/">Indigo</a>). But having a model is all well and good, you eventually want to be able to walk around the model.</p>
<h1 id="fly-throughs">Fly throughs</h1>
<p>Currently I don’t have any collision detection or physics in my 3d world, at the moment you basically fly around the world at ground level, using mouselook and WASD keys. Although this is okay, I think it’d be much more interesting to have proper level physics so you can walk up and down stairs, not walk through walls etc.</p>
<h1 id="bullet-physics">Bullet physics</h1>
<p>I’ve used <a href="http://bulletphysics.org/wordpress/">bullet physics</a> before. One of my early metaverse simulations was a c++ app that ran bullet physics, and reflected the simulation state over websockets, letting you send lua commands to the world to add elements to the simulation. So I took a quick look at ammo.js, which is an emscription compilation of bullet to javascript. My biggest concern with ammo is just it’s size, it’s 1.2mb of uncompressed javascript, and 400kb compressed. I’m guessing that’s going to totally kill mobile devices (which is a platform I don’t want to rule out just yet).</p>
<h1 id="cannonjs">Cannon.js</h1>
<p><a href="http://cannonjs.org/">Cannon</a> is a little pure javascript physics library. It’s pretty cool, and has a few first person shooter demos, including a nice voxel one where you can shoot spheres around the world. The simulation seems pretty fast and reasonably accurate. I suspect it’s not as good as bullet, since it’s a much younger project, but it seems good enough to try out.</p>
<p>The first problem I came up with cannon is that it doesn’t support triangle meshes (see <a href="https://github.com/schteppe/cannon.js/issues/43">this issue</a>). I asked @schteppe about this, and he said that most games end up using convex hulls (convex polyhedrons, sometimes called brushes by game designers) as the collision model, instead of using the mesh directly.</p>
<h1 id="creating-convex-hulls">Creating convex hulls</h1>
<p>The typical way a game designer creates a set of convex hulls to represent level geometry, is to draw it by hand, or manipulate the visual geometry by decimating and simplifiyng the meshes until you have a low res set of hulls to collide against. I want to avoid this if possible, since my target demographic probably aren’t game designers, so I need some automatic way of generating the collision geometry.</p>
<h1 id="hierarchical-approximate-convex-decomposition">Hierarchical Approximate Convex Decomposition</h1>
<p>I found the <a href="http://kmamou.blogspot.co.nz/2011/10/hacd-hierarchical-approximate-convex.html">HACD library</a> while searching for automatic methods to convert meshes into a set of convex hulls. After a 5 minute play around (it was getting late), I couldn’t work out how what parameters to tweak to generate an accurate mesh. For example, this is a model of the barcelona pavillion - there’s way too much approximation going on, making the collision set useless.</p>
<p><img src="/images/hacd.png" />
<cite>Barcelona pavillion processed with HACD</cite></p>
<h1 id="next-steps">Next steps</h1>
<p>The way I see it - there are two things I can do from here. One is to keep playing with the HACD library and try and work out how to more accurately model the 3d models I send for processing. The demos included with the HACD library are pretty impressive, involving lots of complex concave shapes, so I suspect I just have my dimensions, or export process (I’m currently going sketchup to blender to meshconv to .off) incorrect. Anyway, the other option which is equally valid, would be to add trimesh support to cannon.js.</p>
<p>@schteppe said that he doesn’t really believe in triangle meshes, because they have “bad performance and need small timesteps to function properly”, which may be true, but bullet supports trimeshes for static objects, and it’d be a fun challenge adding trimesh support for static objects to cannon.js, so I might have a go at doing that next week. We’ll see.</p>
<p>Either way, my next goal is to have a lightmapped, collision detected model of the barcelona pavillion for people to walk around.</p>
<p>addendum: The last thing I could do would be to write a custom physics library that basically just does collision detection using rays. This is an okay solution, and I could probably reuse a lot of the three.js ray-triangle intersection code, but I’d rather have a proper physics engine to be honest.</p>
My model uploader workflow2014-07-23T00:00:00+00:00http://bnolan.github.com/2014/07/23/my-model-uploader-workflow<p>I’ve been doing quite a bit of work on metaverse.sh (there’s no site there are the moment) lately, trying to get the model uploader workflow going.</p>
<p>One of the things I want to make really easy in metaverse.sh is adding content to the world. So what I’ve got at the moment, is you use first person controls to move around the world, and at any stage, you can drag and drop a collada <code class="highlighter-rouge">.dae</code> model into the world and it’ll get uploaded and appear right in front of you (and will also appear in the world of anyone else connected to the server).</p>
<h1 id="prototype">Prototype</h1>
<p>I prototyped all the bits together, and it worked. Basically, on <code class="highlighter-rouge">dragover</code>, I place a big transparent (input type=”upload”) over the scene. Then when you drop the file, I use a jQuery <code class="highlighter-rouge">$.ajax</code> call to upload the image the the <em>assetserver</em>. The assetserver is a node.js app that takes the upload, does some checks to make sure it’s not too big, then hands the upload off to a ruby script that does more inspection, unzips the upload (collada models require external textures, so you zip the model and textures altogether to upload them), and then fires up blender and runs a script.</p>
<h1 id="blender-script">Blender script</h1>
<p>The blender script deletes all items from the scene, then imports the collada model using blender API calls. It then iterates over all the models in the scene to get a vertex, edge and poly count, and construct a bounding box around the entire mesh. It then uses the threejs exporter script to write out a <code class="highlighter-rouge">.js</code> version of the model. I also write out some data in json format that is passed back to the node.js script.</p>
<h1 id="model-checks">Model checks</h1>
<p>The basic checks I’m going to do on a model are:</p>
<ul>
<li>Polycount - don’t allow models with more than 10k triangles in the main world</li>
<li>Size - autoscale models when added to the world, so they can’t be bigger than 50m on a side</li>
</ul>
<p>I might also do some model fingerprinting so that if people upload material that they want to copy protect, that is possible</p>
<h1 id="back-to-node">Back to node</h1>
<p>The model and it’s textures (converted to .png using mogrify) are then moved to a public folder where they can be hosted, the models data is updated with the vertex count, and the model is saved to a postgresql database. The model is now ready to be served. At this stage, the upload process is finished, and the model data is returned to the client.</p>
<h1 id="add-element-to-the-world">Add element to the world</h1>
<p>The client then calls <code class="highlighter-rouge">addElement</code> over the websocket connection to the server. This adds the model (referenced by id at the moment, since the world server shares a database with the assetserver) to the world, reflects the model to anyone else in the world, and persists the model to the database, so that the model is stil there when the world server restarts.</p>
<p>It’s all pretty hinky at the moment, but I’m enjoying the challenge of making a large javascript project.</p>
Herenow retrospective2014-05-27T00:00:00+00:00http://bnolan.github.com/2014/05/27/herenow-retrospective<p>I’ve been teaching myself mobile development for the past 6 months, and last week, I decided to put aside my full time project and instead try and build and release something quickly, to get experience with the whole app store submission process, and also to write an app with all the knowledge I’d gained over the previous 6 months (my previous project looks like a total nightmare).</p>
<p>What I came up with was <a href="http://herenow.nolanconsul.com/">herenow</a>.</p>
<p>It’s a simple message board app that lets you send push notifications to anyone that’s near you, or to leave a message for the next person to visit where you are. I built it for iOS and Android, with a ruby on rails backend.</p>
<p>A few things I discovered:</p>
<h1 id="i-shouldve-probably-used-heroku">I should’ve probably used heroku</h1>
<p>I deployed the app on my own linode box, since in all likelihood it’ll only be my friends and I using the app. It took several hours of mucking around to get my linode up and hosting the rails app. The only real positive is that I can leave the app running for months and months without paying $100 a month for a heroku worker + postgres database.</p>
<h1 id="ios-is-easier-to-make-pretty">iOS is easier to make pretty</h1>
<p>I wrote the app for iOS first, and (this might be contentious but) I found the iOS dev tools better for writing a good looking app. Once I’d designed the app, I found it easy to copy the design over to Android, but out the gate, I’ve found my android apps are uglier.</p>
<h1 id="gcm-is-way-easier-than-apn">GCM is way easier than APN</h1>
<p>The APN setup sucks, fucking around with certificates and shit, it’s totally bogus.</p>
<h1 id="intellij-is-great">Intellij is great</h1>
<p>Once I worked out how to sync my gradle build scripts with the intellij studio (hint, click the refresh button on the gradle pane), I found intellij great. I didn’t really have any problems at all adding a bunch of dependencies and coding against them.</p>
<h1 id="just-getting-released-is-exhausting">Just getting released is exhausting</h1>
<p>So I mocked up the iphone app in an evening, then spent the weekend getting the backend service running and APN working, then monday I wrote the android app and today I polished everything up a bit (it’s all pretty hinky tbh). It’s exhausting just getting to here. I can’t imagine how difficult it must be for single developers who are trying to build and release and market apps. You’re pretty much knackered by the time you get to release something, and that’s the point that you have to have heaps of energy to fix bugs, maintain servers and do marketing.</p>
With - Quantified Self App2014-05-20T00:00:00+00:00http://bnolan.github.com/2014/05/20/with---quantified-self-app<p>So I’m getting closer to releasing With. I was reading <a href="https://signalvnoise.com/posts/3743-hybrid-sweet-spot-native-navigation-web-content">DHHs post</a> about hybrid apps, where you use native navigation and display content using webviews, and since it was slow going trying to write a purely native version of With, I decided to go hybrid.</p>
<p>This made the app way simpler. Instead of a signup form, encounters list and friends list, it’s now just a segmented control to change tabs and a webview. I quickly hacked it up for iOS and got most of the app working, and then refactored the Android app. The app is way simpler now. It’s kind of annoying working on a new mobile app, I’m so new at mobile development, that I spend the whole time learning. At least, in a positive way, once I’ve finished making With, I should be much quicker at building future mobile apps.</p>
<h1 id="hosting">Hosting</h1>
<p>I’m hosting With on Heroku, and the way the encounters detection works is super un-optimized, so I’ll probably only release With in New Zealand for starters, since even having a 1000 users will probably require that I rewrite the encounter detection code. I’m trying not to think about scaling the app, since the most likely outcome with any app is that no one will use it, rather than tens of thousands of users. But yeah, if I do get more than just my immediate friends using the app, I’ll need to refactor the hosting. I’ll probably just move everything to node.js, ec2 and rds (since you can use <a href="http://blog.safe.com/2013/12/postgis-on-aws-rds/">postgis on RDS</a>).</p>
<h1 id="ios-and-android">iOS and Android</h1>
<p>I’ve been trying to keep my function signatures the same across iOS and Android, so I can <code class="highlighter-rouge">[self showEncounters]</code> or just <code class="highlighter-rouge">showEncounters()</code>. It is interesting how much easier iOS seems than Android, because it feels more like javascript, where most tasks are async, and you just get a callback when the task is complete.</p>
<h1 id="hashed-phonebook">Hashed phonebook</h1>
<p>With hashes and uploads the phone numbers in your phonebook to the server, so that the server can look for your friends using the app. I’m a little uncomfortable about this, since generating a rainbow table for 12 digit phone numbers is trivial on a modern GPU. It’s not too bad, because I’m not uploading the names in your phonebook, so if an attacker does get access to the database, they’ll just know who you call, not what their names are. I think the best security I can do here is to get rid of the hashed phone numbers off the server after x days, instead of holding onto them forever. I guess I can also hash together the phone number of user a and user b together, which will make generating a lookup table slightly (but not much) harder.</p>
<h1 id="encountersinvolvingself">Encounters.involving(self)</h1>
<p>When I initially pitched With, I thought I would show you all of your friends encounters, so that if Sam and Dave are hanging out together on a tuesday after work, I can send them a text and ask what they’re up to. But as I’ve developed the app further, I’ve realised that your newsfeed is actually very personal (for example it shows when you arrive and leave from work), so I think for the first release, I’ll only show encounters that you personally were involved in.</p>
<h1 id="addresses">Addresses</h1>
<p>I’ve decided that I’ll show the street that an encounter took place on. So it won’t say that you were at Hashigo Zake, it’ll say you were on Taranaki Street. I might even add the nearest street intersection, since that’ll place you pretty well. I don’t think automatic picking of locations using the foursquare venues api will be accurate enough, and it’s better to give a vague address than an incorrect venue name.</p>
<h1 id="grinding-out-the-release">Grinding out the release</h1>
<p>I’m pretty sure once I’ve finished all my learning and experimenting, I’ll get to the point where I could rewrite the entire app in a few days, but it’s been an interesting learning experience. Now it’s just a case of grinding out the app until I can release on the app stores to my friends and colleagues in New Zealand.</p>
Reddit metros2014-05-13T00:00:00+00:00http://bnolan.github.com/2014/05/13/reddit-metros<p>I think I’ve posted about this before, but having had a bit more experience building mobile apps, I think there’s a great opportunity to take the list of <a href="http://www.reddit.com/r/LocationReddits/wiki/index">local reddits</a> and build a customised app with a messageboard for each city. Basically replicate the post link, upvote/downvote, start conversations functionality that reddit provides on a city basis and make a customised app for your city that does this.</p>
<h1 id="app-per-metro">App per metro</h1>
<p>I use the term metro here to discuss a city, or a metro area. I’m not enough of an expert in app store marketing to say whether making an app per metro is a good idea. On the plus side, when people install the app, you can automatically connect to other people that have a location in common with them. You also don’t have to use the geolocation API to find out where they live. People might also feel more connected to the app since it’s an app for <em>my</em> city. On the negative side, the process required in maintaining a seperate app for hundreds of cities would be massive, and Apple might even reject you from the app store for creating so many spammy apps.</p>
<h1 id="cordova-vs-mobile-web">Cordova vs mobile web</h1>
<p>I think you want to do this as a cordova / phonegap app, or maybe even a proper hybrid app, so that you can do push notification when someone replies to your topic. I think a mobile web site isn’t sticky enough to compell people to use it, you kind of want to take up space on someones homescreen. You’d probably also want to be able to use Android intents (and the iPhone equivalent) to redirect people to maps links, to open place reviews in yelp, etc.</p>
<h1 id="integrated-w-4sq-yelp">Integrated w/ 4sq, yelp</h1>
<p>When people ask a question in the app like: “What places do good gluten free pancakes”, it’d be nice for people to be able to post a reply as a ‘place’, ‘link’ or ‘text’. If they write a place name, you should search yelp and foursquare so that people can view the mentioned place on a map. Maybe even have a map view of a topic, showing all mentioned places on a map, so that you could have one super topic which was “great grafitti spots”.</p>
<h1 id="voting--moderation">Voting / moderation</h1>
<p>I think something that works really well in Reddit, is that you have pro-active moderators in metro areas. I think you’d need to build moderation tools, because community upvoting / downvoting wouldn’t be enough. You’d also want to attract some moderators and encourage them to post topics when the app for the metro was first released.</p>
<h1 id="one-day-project">One day project?</h1>
<p>I like this project, it feels simple enough that you could knock up a prototype in one day, and then market it in your own city and see if you could make something fun. No scaling issues, relatively simple UI, no fancy background processes.</p>
Better encounter detection2014-04-29T00:00:00+00:00http://bnolan.github.com/2014/04/29/better-encounter-detection<p>So I got as far as making a <a href="http://bennolan.com/contour-terrain/html/">cool contoured terrain</a> in WebGL before I decided that it was time to do a bit more work on With. I downloaded the latest database dump from Heroku and started doing some analysis. Where I got to last with With, is that I couldn’t reliably detect encounters between people, so I put it on the backburner while I worked on other stuff and talked to friends about was to better detect encounters.</p>
<h1 id="using-st_buffer">Using st_buffer</h1>
<p>Every location fix I get from a user has a latitude, longitude and accuracy. Krasimir asked me why I don’t use something like st_buffer to create a circle around the location fix, and see if that circle intersects with any other users <code class="highlighter-rouge">fixes</code>. This was actually a really cool idea, so I refactored my encounter simulation code like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>create temporary table
current_locations
as
select
id,
geo as geo,
st_buffer(geo, accuracy) as fix,
created_at,
user_id
from
locations
where
created_at > #{conn.quote(earlier_t)}
and
id in (select max(id) from locations where accuracy < 500 and created_at < #{conn.quote(t)} group by user_id);
</code></pre>
</div>
<p>This way, instead of searching for all encounters within an arbitrary distance of each other, I use the accuracy of the fix to see if the two circles intersect using st_intersects.</p>
<h1 id="using-intersections-to-define-the-encounter">Using intersections to define the encounter</h1>
<p>The next thing, was defining where about the encounter took place, so that you can compare old encounters to new encounters (since every 5 minutes I either extend an encounter and say it’s still in progress, or I end date it and create new encounters for wherever you are now). I had a few ideas, first up was to use the centroid of a union of the two buffers, but then I thought about it, and realistically, the encounter should have happened in the intersection are of the two buffers. This may create weird lens shaped intersection areas if the two users two locations are quite different, but in theory the intersection area should cover where the encounter actually occurred.</p>
<h1 id="two-person-encounters-or-multi">Two person encounters or multi?</h1>
<p>The next problem I was working on, was how to create multi person encounters. You have two problems, people leave and join encounters at different times (eg I might be at the pub with Sam, and Patrick might turn up half an hour later), and also, you have to consider friendship networks. For example, I might be at the pub with Robin, and Robins friend Dave is there. Does this mean that I’m in an encounter with Dave? I don’t know. What if Dave has a friend who I don’t know - how greedy do the encounters get? Do they gobble up everyone who has a friendship with anyone else at the location - I don’t think so, but I couldn’t work out a nice way to represent this.</p>
Positional audio2014-04-17T00:00:00+00:00http://bnolan.github.com/2014/04/17/positional-audio<p>A big part of the VR experience (especially in a headset like the oculus rift) is the ability to talk to other users. With the rift on, you can’t see the keyboard, and text overlays look a bit manky anyway (since DK1 is so low resolution), so being able to naturally talk to people is a great idea. So I’d like to have voice chat in my scene-server project.</p>
<h1 id="everyone-always-connected">Everyone always connected</h1>
<p>The brute force way of making this work is making everyone constantly connected to a mixing server that mixes the channels for each avatar and streams it down to your PC, so your client has only one inbound and one outbound connection, and the server does all the mixing. Obviously this will be super expensive for the server, since you’ll be processing dozens or hundreds or thousands of connections and then mixing them together. Even if you limit channels so you can only hear anyone within 20 metres, it’ll still be expensive, so not keen on this way.</p>
<h1 id="auto-connect-to-anyone-nearby">Auto connect to anyone nearby</h1>
<p>Using webRTC and connecting to each other via a STUN server, you can do a direct p2p connection to other clients in the world. This way, your client could just choose the 3 nearest people and auto connect to them, and anything they say you could hear. This is a pretty nice solution since it’s relatively automatic.</p>
<h1 id="ui-for-click-to-talk">UI for ‘click to talk’</h1>
<p>This is currently my preferred way, in that you walk up to someone, and if you click on them, it’ll start a webRTC connection with them and you can start talking to each other, and the connection will last until you’re far enough apart to stop talking.</p>
<h1 id="positional-audio">Positional audio</h1>
<p>This is something I haven’t been able to find any prototypes of yet, but I’m pretty sure is doable. It’s using this <a href="http://www.html5rocks.com/en/tutorials/webaudio/positional_audio/">positional audio</a> article combined with webRTC so that you can sample the microphone, send it over to another client and then play it back in stereo and faded for the distance away the other person is, to help with immersion. I’m looking forward to having a hack at this.</p>
Dom Mutators2014-04-11T00:00:00+00:00http://bnolan.github.com/2014/04/11/dom-mutators<p>What I’m envisaging for the scene server I’m working on at the moment, is a dom-like API that you can script against, and then changes to the world get reflected down to all the connected clients. So you’ve got something like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><box position='50 10 0' rotation='0 1.57 0' scale='1 1 1'>
<script>
{
tick : function(self){
var t = new Date().getTime() / 10000.0 + 20
self.setAttribute('position', new THREE.Vector3(Math.sin(t * 11) * 200,10,Math.cos(t * 7) * 200));
}
}
</script>
</box>
</code></pre>
</div>
<p>That’s a first cut, the end code will probably be quite different to that, but as you can see, you can animate a box (moving around in a circle) using just a few lines of code, and using dom methods like <code class="highlighter-rouge">setAttribute</code>. The current way the code works, you actually have to call <code class="highlighter-rouge">setPosition</code>, which marks the node as <code class="highlighter-rouge">dirty</code>, so it’s position is sent to all the connected clients at the next tick.</p>
<p>This is all well and good, but I’m looking to generalize the code a bit. In the long term, I’m sure there will be lots of need for performance, but for starters, I’d like to off load as much of the api writing and scene parsing to other libs. So one of the things I’ve been thinking of, is using something like the <a href="https://developer.mozilla.org/en/docs/Web/API/MutationObserver">dom mutators api</a> to keep track of changes to the dom, and then reflect them out.</p>
<p>It’d make my code a lot simpler. Something like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>networkTick: ->
packets = for node in @changes
new Packet.NotifyChanges(node)
send(packets)
@changes = []
onMutationEvent: (node) ->
@changes.push node
</code></pre>
</div>
<p>Obviously it’d be a bit more complicated than that, for starters you want to use the <code class="highlighter-rouge">UpdatePosition</code> packet which binary encodes positional changes and sends them over the wire, rather than serializing the entire node. But I think it’s a nice way to get started. Sadly - <code class="highlighter-rouge">dom-lite</code> doesn’t have built in support for the MutationObserver api. But that might be a task for this weekend, adding support for the MutationObsever api to dom-lite. We’ll see…</p>
Parsing scene files using dom-lite2014-04-10T00:00:00+00:00http://bnolan.github.com/2014/04/10/parsing-scene-files-with-dom-lite-and-htmlparser2<p>So, I went to <a href="http://zoomin.co.nz/wellington/newtown/rintoul+street/4/-monterey/">Monterey</a> with Rissa last night and drunk a few beers and had a hack at using <a href="https://www.npmjs.org/package/dom-lite">dom-lite</a> to provide a dom interface to my scene nodes. While I was working on it, I came across <a href="https://www.npmjs.org/package/micro-dom">micro-dom</a> which uses <code class="highlighter-rouge">htmlparser2</code> to parse html and turn it into dom-lite nodes. So I decided to replace my handwritten scene parser with htmlparser2.</p>
<p>The full code that I used to prototype the ideas is in this <a href="https://gist.github.com/anonymous/10329377">gist</a>. First up I created a class Element, that extended from HTMLElement, and I could add my own getters, setters and other handy methods too.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class Element extends HTMLElement
@getter 'innerXML', -> @innerHTML
@getter 'outerXML', -> @outerHTML
@setter 'innerXML', (xml) -> @_setInnerXML(xml)
@setter 'outerXML', (xml) -> @_setOuterXML(xml)
</code></pre>
</div>
<p>Because I’m working with XML (or SceneML, or whatever I want to call the scenefile descriptions), I proxy the innerHTML methods and call them innerXML. Something that mv-server has to do quite a lot, is parse xml for individual scene nodes, so I decided I could do this using the outXML method. Something like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if packet.type == Packet.XMLChanged
scene.getElementById(packet.id).outerXML = packet.xml
</code></pre>
</div>
<p>I implemented this like so…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>_setOuterXML: (xml) ->
parsed = (err, nodes) =>
throw err if err
throw 'Too few/many nodes' if nodes.length != 1
node = nodes[0]
throw 'Dont do that' if node.name.toUpperCase() != @nodeName
@childNodes = []
@attributes = []
for name, value of node.attribs
@setAttribute name, value
addChildren(this, nodes[0].children)
parser = new htmlparser.Parser(new htmlparser.DomHandler(parsed))
parser.write(xml)
parser.end()
</code></pre>
</div>
<p>This uses the htmlparser to populate the dom node. Once I had tried out most of the ways this could work, I started hacking up mv-server to use dom-lite and htmlparser2 as a backend. I didn’t manage to finish it last night, but I’ll have a go this evening possibly, and see if I can get the world server to work. One of the advantages, is that I can program against a normal DOM model, and the loader will also handle comment nodes, text nodes will be handled properly, etc. The biggest problem I’ve had so far, is that everything that gets sent over the wire has it’s <code class="highlighter-rouge">id</code> attribute stored as a <code class="highlighter-rouge">DWORD</code>, and dom-lite seems to cast all attributes to strings, so I have to cast those id attributes back into an integer before I can encode them to be sent over the wire.</p>
<p>All very interesting anyway. I’m really enjoying the <code class="highlighter-rouge">npm</code> ecosystem.</p>
Tweening with obelisk2014-04-08T00:00:00+00:00http://bnolan.github.com/2014/04/08/tweening-with-obelisk<p>While working on the isometric viewer for <code class="highlighter-rouge">mv-world</code>, I added some basic tweening to obelisk using tween.js.</p>
<p>You can see the code <a href="https://github.com/bnolan/isometric-test/blob/master/app/src/packets.coffee">here</a> in the packet processor. I’m pretty sure this is the wrong place to be doing animation, but basically the idea is…</p>
<h1 id="2-hertz-network-updates">2 hertz network updates</h1>
<p>So we get two network positional updates per second. This is currently hard coded, but in the future, should be calculated by measuring the number of updates per second (as the server may throttle you if there’s too much going on, or your network connection is a bit rubbish).</p>
<h1 id="store-positional-data-as-a-threevector">Store positional data as a THREE.Vector</h1>
<p>I’m trying to write the scene / node code in a way that can be extracted into a bower / node module for creating headless, isometric, text-only and 3d clients for the mv-server. To that end, I’m storing everything as a three vector.</p>
<h1 id="using-tween-with-threevector">Using Tween with THREE.Vector</h1>
<p>Tween doesn’t support using the Vector3 class directly, so I have a little <code class="highlighter-rouge">onUpdate</code> helper that converts an object with {x,y,z} attributes into a proper vector, that is then set on the scene element.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>tween.to( { x : @positionX, y : @positionY, z : @positionZ }, 500).
easing(TWEEN.Easing.Linear.None).
onUpdate( -> element.position = new THREE.Vector3(@x, @y, @z)).
start()
</code></pre>
</div>
<h1 id="render-loop-reads-from-the-scene-element">Render loop reads from the scene element</h1>
<p>Obelisk doesn’t have it’s own scene graph, you just call <code class="highlighter-rouge">pixelView.renderObject</code> on each element. It doesn’t even actually do it’s own depth sorting, hopefully that’ll be added. So for the render loop, I just iterate over every element in the scene and call render, converting the three.js vector into a obelisk pointer. I had some performance issues originally, but after I talked to Max (the obelisk maintainer) he told me not to create the obelisk objects in the render loop and now I can render several hundred elements at 60fps on my macbook.</p>
Using blender to convert models2014-04-07T00:00:00+00:00http://bnolan.github.com/2014/04/07/using-blender-to-convert-models<p>So I got the <a href="http://github.com/bnolan/mv-server/">world server</a> working for my 3d world project. You can reliably connect from the client which is currently using obelisk.js to render an isometric overview of the world. I also added <a href="https://github.com/bnolan/mv-server/blob/master/scenes/hello.xml">script</a> element to the scene, so you can script an animation on the server, and it’ll get replicated out to all the connecting clients.</p>
<p>I don’t have any online demos for this stuff yet, because I can’t decide on what domain name I want to use for this. The current architecture I’m thinking of:</p>
<h1 id="nodejs-world-server">Node.js world server</h1>
<p>An opensource world server that runs in plain (with optional native models for linux servers) that anyone can run. It loads a <code class="highlighter-rouge">scene.xml</code> scene description, and can serialize it’s state back out to xml at any stage when you want to save out the current world state.</p>
<h1 id="isometric-world-viewer">Isometric world viewer</h1>
<p>This viewer uses obelisk (or something similair) to view the world. You don’t have an avatar in this world, but you can authenticate with the world server and modify the world. It’s basically a minimal world for testing the client side viewer code. The client code will be shared with the three.js viewer.</p>
<h1 id="threejs-viewer">Three.js viewer</h1>
<p>A fully featured viewer that uses three.js to provide a first person experience in the world. Will have oculus rift, positional audio and webgl support.</p>
<p>The javascript for both the viewers will run on a central server, so that you can embed world viewers on your webpage, in a similar way to how you can use the google maps api anywhere.</p>
<h1 id="asset-server">Asset server</h1>
<p>This will run on a similar server, and will allow uploading of many different types of models, convert those models into a three.js compatible model format, and upload the models to a CDN so that world servers can quickly refer to the models. This server will use various binary tools to convert meshes, sanitize and rasterize them. To this end, I started writing some blender scripts to import different model formats, render an image of them, rescale them so they are 1 unit in size in the longest dimension (I’m picturing a world where all the models are normalized to the same size, and you specify a scale parameter when you add the model to the world to set the size of the model), and then output the 3d model file.</p>
<p>Out of the box blender seems to support <code class="highlighter-rouge">collada</code>, <code class="highlighter-rouge">3ds</code>, <code class="highlighter-rouge">obj</code>, <code class="highlighter-rouge">stl</code>, <code class="highlighter-rouge">x3d</code> and obviously <code class="highlighter-rouge">blend</code> files. I haven’t finished my normalizer / converter / importer, but there is a <a href="https://gist.github.com/bnolan/10016675">gist</a> of what I’ve managed so far. The blender api isn’t very well documented. :(</p>
Scene server2014-04-04T00:00:00+00:00http://bnolan.github.com/2014/04/04/scene-server<p>So the last few weeks I’ve been experimenting with various multi-user 3d environment things. I originally started with a c++ app that uses bullet physics and websockets to provide a shared 3d world where anyone can drop blocks into the world and they physically interact, and the results are streamed out to all the connected clients and rendered using three.js.</p>
<p>At some point I decided to <a href="https://github.com/bnolan/mv-server">rewrite everything in node.js</a>, for ease of hacking, but keep using technology that could be rewritten in c++ at some point if I decided to. The current goal is to build something where:</p>
<ul>
<li>you can see other avatars in the world with you</li>
<li>server admins can upload collada .dae art into the world</li>
<li>anyone can host their own node server</li>
<li>there is a central oauth server for verifying users user names</li>
</ul>
<p>I’m not sure all of these are important, and it’s missing the thing that I most wanted, which was to let anyone be able to drag and drop a collada model into the world, but I haven’t worked out all the permission stuff yet about how that would work (for example, you’d have to have moderators to delete all the penis models, and how do you prevent people dropping a huge box over the entire world so that you couldn’t see anything?).</p>
<h1 id="node-server">Node server</h1>
<p>The node <a href="https://github.com/bnolan/mv-server">world server</a> is written using mostly pure javascript modules, so that in theory it can be run on windows boxes without a cygwin build environment. It’s currently got a handful of tests, and is running on travis-ci.</p>
<p>The idea behind the world server, is that you write a scene in xml, which describes all the 3d models in the scene as <a href="https://github.com/bnolan/mv-server/blob/master/lib/node.coffee">nodes</a>, where they are, what their rotation and scale is, and then fire up the server. The server loads the scene file, then listens for websocket connections. Anyone who connects to the server then sends up their current location in the world and an <code class="highlighter-rouge">Observer</code> object is created. The server then sends any objects within the observers visible radius to the client.</p>
<p>The initial <a href="https://github.com/bnolan/mv-server/blob/master/lib/packets.coffee#L1">introduction</a> of an node to an observer is by sending the xml description of the node. After the initial introduction, the observer only gets updated if a node moves in the world (maybe it’s subject to physics, or is being animated by a script in the world). These update packets are a lot smaller, since they only include position and rotation squashed down.</p>
<p>All the messages are encoded via msgpack.</p>
<h1 id="viewers">Viewers</h1>
<p>I’m currently working on a basic <a href="https://github.com/nosir/obelisk.js/">obelisk.js</a> powered client that just shows the location of all the boxes and models in the world, as a debugging tool and example of a minimal client. The real target however, is to port some of my three.js-powered viewers over to the new protocol, so that you can have an oculus-rift-capable viewer for 3d scenes that are served via the node server.</p>
<h1 id="avatars">Avatars</h1>
<p>The next thing to add will be avatars, so that you can see other people experiencing the world at the same time as you. Then we’ll have a proper multi-user 3d space which will be cool.</p>
Wire formats for the metaverse2014-04-02T00:00:00+00:00http://bnolan.github.com/2014/04/02/wire-formats-for-the-metaverse<p>I’ve been spending my free time working on some metaverse style projects. This has been kicked off by two things:</p>
<ul>
<li><a href="http://indigorenderer.com/">Nick</a> visiting for a few weeks and talking about the metaverse over beers</li>
<li>Borrowing <a href="http://mindscape.com/">JDs</a> oculus rift and seeing what VR might become</li>
</ul>
<p>So I started off by writing a little word simulator in c++, using the bullet physics engine and lua, and a client using three.js that connected to the world and saw a big shared virtual world. This was all pretty cool, but I’m not a proficient c++ developer, so I decided to try hacking up a world server using node.js, since it’s easier for other people to hack on, and it’s a fast prototyping language.</p>
<h1 id="a-wire-protocol">A wire protocol</h1>
<p>I’m using websockets as the communication layer, and I was trying to come up with a wire protocol that’s nice and fast for representing the movement of a bunch of avatars and physics-simulated blocks in realtime. I was looking at doing 10fps from the server, and having the client interpolate these.</p>
<p>The simplest data format (a packet representing movement of a 3d object) is:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="err">id</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">uint32</span><span class="w">
</span><span class="err">positionX</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">float64</span><span class="w">
</span><span class="err">positionY</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">float64</span><span class="w">
</span><span class="err">positionZ</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">float64</span><span class="w">
</span><span class="err">eulerX</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">uint8</span><span class="w">
</span><span class="err">eulerY</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">uint8</span><span class="w">
</span><span class="err">eulerZ</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">uint8</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>I decided to express rotation as three euler angles compressed into a byte, using 3 bytes, instead of 4 * float32s of a quaternion, since its easy to convert eulers to quaternions in three.js, and I saw that the minecraft protocol does this also. In retrospect, i’d probably encode the eulers as uint16s, to give a bit more accuracy on larger models, but who knows. I’d also prefer to use float32s for the position, since my worlds are all going to be quite small and you’ll always be near the origin, but nick recommended float64’s, and that’s what msgpack seems to use by default so it’s ok by me.</p>
<h1 id="json">JSON</h1>
<p>First up, I tried encoding the movement packets as json, which is pretty fast, but super space inefficent (since you’re sending floats as strings).</p>
<h1 id="custom-encoding">Custom encoding</h1>
<p>Then I wrote a custom encoder / decoder using dataviews that writes out a binary representation of the packet. This is fine, except it means you have to write customer encoder / decoders for each packet type that you have, plus with the fixed-sized packet format I was using, it wasn’t possible to encode strings, which would mean falling back to json for any packet with a string embedded (which might be a model url, message from one user to another, etc).</p>
<h1 id="msgpack">Msgpack</h1>
<p>I wrote my custom encoder, thought it was pretty cool, and was just about to turn it into a npm module, when I decided to have a quick look at what other binary wire protocols there are, and I found msgpack. Msgpack is pretty cool, it converts a javascript object (or array in my case) into a binary representation that is quite efficient. It seems to use float64s for floats, and bytes for values in the range 0-255.</p>
<h1 id="performance">Performance</h1>
<p>After reading a bit, I had heard bad things about the performance of msgpack, so I decided to write a <a href="https://gist.github.com/bnolan/9926889">script to test the performance</a> of the different encodings.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Json...
* benchmark took 7 seconds and 35 ms
* average size of packet stream is 260097
Msgpack...
* benchmark took 5 seconds and 209 ms
* average size of packet stream is 125770
Custom...
* benchmark took 7 seconds and 312 ms
* average size of packet stream is 106240
</code></pre>
</div>
<p>Unsurprisingly, my custom encoder was the most concise format on the wire, since it just writes out a two-byte packet header and then all the fields. JSON was the largest. But the biggest surprise was that msgpack (using the node module which calls out to c) was the fastest to encode the packets, as well as having a wire format that was only 20% larger than my custom method.</p>
<h1 id="performance-in-the-browser">Performance in the browser?</h1>
<p>I haven’t tested performance in the browser, but I’m guessing that msgpack won’t be as efficient there, since it doesn’t have a native module, however it packs the messages into byte-sized chunks, and doesn’t have to do individual bit twiddling to get the message out, so I think performance should be fine. It’s the server that’s really going to struggle if it has multiple concurrent users anyway.</p>
<p>So I think i’ve decided on msgpack.</p>
<h1 id="windows">Windows</h1>
<p>I’m still undecided about this, but I thought it’d be cool to write the world server in node in such a way that it can be run on windows boxes without requiring a cygwin environment, ie use native javascript modules instead of binary ones. Msgpack has a pure javascript implementation, and i’m using xmldom instead of libxml. This means that if windows users won’t to try creating their own 3d worlds and serving them up via node, they are able to (without having to fire up a linux vm).</p>
A virtual noticeboard for your pub2014-01-21T00:00:00+00:00http://bnolan.github.com/2014/01/21/a-virtual-noticeboard-for-your-pub<p>This is a pretty niche app, and I think it’d be hard to grow, and people probably wouldn’t like that it always runs in the background and uses your battery, but here’s the idea.</p>
<h1 id="background-checkins">Background checkins</h1>
<p>The app runs in the background all the time, and when it detects you are at a new location, it checks you in there. The check in isn’t shared with anyone, but it means that the server knows where you are.</p>
<h1 id="post-messages-to-your-location">Post messages to your location</h1>
<p>Then, once you’re checked in somewhere, you can post a message to a ‘virtual messageboard’ that is your current location. You can only see this messageboard if you’re at the location, and you can only post if you are at the location as well.</p>
<h1 id="send-notifications-to-people-at-your-location">Send notifications to people at your location</h1>
<p>When a message is sent to the location messageboard, everyone who is checked in gets a google cloud messaging notification saying ‘xyz has posted a message’. You can then reply to the message, or write your own message onto the messageboard.</p>
<h1 id="the-big-daddy-is-super-fresh">The “Big Daddy is super fresh!”</h1>
<p>You could use this post messages about the quality of the beer, to complain about someone farting, or maybe write some comments about how killer the band is, or you could use it to message other people in your housing complex. Maybe if you could send stickers out to venues, whenever people start messaging each other there, saying ‘there’s a conversation about this place in a parallel universe, install the app to see what people are saying’</p>
<h1 id="unique">Unique?</h1>
<p>There’s been a million apps that allow location-based messaging, but none of them have taken off. I wonder if the fact that this isn’t location based messaging that visible to everyone, but instead a location-specific messageboard that you can only see when you’re there, I wonder if that’s enough of a difference to make this a goer?</p>
<p>Another android app to put on my backlog.</p>
Foursquare times snapchat2013-12-11T00:00:00+00:00http://bnolan.github.com/2013/12/11/foursquare-times-snapchat<p>Sorry, this is a bit of a puff piece, but it’s been interesting seeing the large social networks (Facebook) being stripped down into their component parts and have their lunch eaten by small task-focussed mobile apps (like Snapchat).</p>
<p>Me personally, I’m really interested in location-based apps, and I wonder if there’s a market to take the foursquare mechanic and re-make it a way that’s more interesting for teens. I’d include the ability to tag other people, so you could say “Ben and Sam are at Sams house”, and your friends would see the update and could comment on it. You could gamify it by saying “Ben and Sam have hung out every week for x weeks”. I’m not sure if it’s a positive mechanic or a negative mechanic, ie, would people use it to re-enforce their social bonds, or would they use it to bully other kids “Haha Mary, everyone else is at Dorothys except you”.</p>
<h1 id="security">Security</h1>
<p>I’d want to be quite particular with developing an app like this, since you’re holding lots of relatively sensitive data (the locations and hangouts of young people), you don’t want to move fast and break things in a way that leaked lots of data.</p>
<h1 id="foursquare">Foursquare</h1>
<p>This isn’t a natural market for foursquare to get into. Foursquare does have private venues, but they’re not a focus of the app. It’s more about public venues and recommending them to you. The way young people hang out, they don’t tend to use foursquare venues as much (maybe public facilities like sports grounds and swimming pools, but not so much cafes and bars), and I guess they spend a lot more time at friends places. Foursquare doesn’t have and can’t gather this data, so it’s not really going after them.</p>
<h1 id="ephemerality">Ephemerality</h1>
<p>The ephemerality of snapchat is a great feature of it, it means that you have to constantly use the app to get content. It’s also fun to think of how you can build an app with similair functionality, because you [a] have less app that you have to build (no profile pages, no history pages, just current content) [b] have to work with the constraint that people will be constantly adding new content.</p>
<p>I guess the foursquare feed already is pretty ephemeral, you could just show who has been active in the app in the last 24 hours. A good way to gather lots of data might be to run background location gathering, so that if your location has changed in the last hour, poll again in 10 minutes. If the location is still the same, you’re probably at a new location, prompt the user whether they want to share this with their friends, or even do it automatically. Maybe have an ‘off mode’ for when you don’t want to share your location. If this is a location you’ve been at before, allow friendly names (eg you’re at 22 Bottomly Road - “Sams house”).</p>
<h1 id="blank-state">Blank state</h1>
<p>I’m not sure what the ‘blank state’ of this app would look like, and how you’d get people to start using it. I’m not sure about the tagline ‘share your location with your friends!’ - I don’t think that’s what would drive use of the app, maybe it would, I don’t know. Maybe something more around ‘play a game about location with your friends’. Something. And add friends by everyone in your phone book. Maybe allow SMS invites to your friends if you want to expressly share your location with them.</p>
<p>You could call it snapsquare. Hahaha. ;)</p>
<p>ps - A quick search found <a href="http://www.forbes.com/sites/kashmirhill/2013/06/26/glympse-is-snapchat-for-location-sharing/">Glympse</a>, but that’s not really what I was thinking about.</p>
Everythings coming up mobile2013-11-29T00:00:00+00:00http://bnolan.github.com/2013/11/29/everythings-coming-up-mobile<p>This isn’t a particularly controversial position, but I feel that in my own career, now is the time to learn some proper mobile development. I feel that in about two years time, people will do most of their casual browsing on a mobile device, and most of that will be through mobile apps, not through mobile websites.</p>
<p><img src="http://media.giphy.com/media/ZORu2ILrunb1e/giphy.gif" />
<cite>Everything is coming up milhouse</cite></p>
<p>I think people will still use desktop computers or laptops for their day job, or if they need to do admin work, but I think most casual research and play and surfing will be on mobile apps. I may be wrong, but it just seems like that’s the way things are going, I spend more and more of my at-home time playing with apps on my mobile phone instead of using a browser.</p>
<p>So, in case the web has had it’s day, I think I’ll try my hand at more mobile development, in case that is the way the tide is turning.</p>
Post Google Search2013-11-26T00:00:00+00:00http://bnolan.github.com/2013/11/26/post-google-search<p>I’ve been thinking a bit lately about search in the post-web world. It looks more and more likely that mobile apps will become as popular as the web was. The web is never going to go away, but you’re going to spend more than half your time looking at content in a mobile app. So is there a market for a search engine that only shows results that link to apps on your phone?</p>
<h1 id="googles-deep-linking">Googles deep linking</h1>
<p>Google has been working on adding app deep linking to their search results, so if you search for a recipe, and you have a recipe app on your phone, it displays a button next to that recipe, that launches the native app on your phone. It works by registering a web intent in your android manifest, and adding a <code class="highlighter-rouge">link</code> tag to your html, so that when google crawls your site, it knows how to open that content in your native app.</p>
<p>This is a pretty good solution, and only just launched, so it’ll be interesting to see how it grows. There are a few problems with it, but they’re all solvable. For example, it doesn’t prompt you to install an app if you find content that you don’t have the app for. It also displays both web and native content, you can’t tell it to only show links that you have the app installed for. All pretty easily solvable. It doesn’t do one thing though, and that’s personalised indexing.</p>
<h1 id="greplin">Greplin</h1>
<p>Greplin was a startup that came out of ycombinator. It’s an interesting story, the founder started with ycombinator, worked on his idea for 2 months, realised it was useless, then came up with a new concept two weeks before demo day, nailed the prototype and raised a bunch of funding based on his 2-week-old prototype. What Greplin did, was you signed into Greplin, then authorised a bunch of APIs (gmail, linkedin, facebook, etc), and Greplin would crawl all the content on those sites and add it to a personalised search engine just for you. It’s a great idea, you can search all your own content online, from one spot. Sadly, it doesn’t seem like this was a killer idea, because Greplin morphed into cue (a sort of google now product), that was eventually bought by Apple, and shut down. So I’m guessing they never got the huge amounts of traction to become a valid product on their own.</p>
<h1 id="merging-the-ideas">Merging the ideas</h1>
<p>I wonder if there’s room for an android app that you install, it gets a list of all your installed apps (using (getInstalledPackages)[http://stackoverflow.com/questions/7374704/getpackagemanager-getinstalledpackages-packagemanager-get-activities-return]), and then uploads that list of apps to a central server. Users can then vote up the apps they’d like to see indexed by your search engine. You then write an integration or point a crawler at the mobile website of the app, and work out how to deep link the content back to intents in said app. Users could also use oauth to authenticate themselves, and you could generate a personal index of their content, with a nice little android widget for searching for that content. A quick glance at my phone shows the following apps that I’d probably search across if it was possible:</p>
<ul>
<li>Baconreader (reddit)</li>
<li>Facebook</li>
<li>Evernote</li>
<li>Gmail</li>
<li>Goodreads</li>
<li>Foursquare</li>
<li>Instagram</li>
<li>Gmail</li>
<li>Keep</li>
<li>Maps (probably impossible to index)</li>
<li>People (impossible to index?)</li>
<li>Pirate bay browser</li>
<li>Ratebeer</li>
<li>Spotify</li>
<li>Trade me</li>
<li>Tumblr</li>
<li>Twitter</li>
<li>Untappd</li>
<li>Vine</li>
<li>Yelp</li>
<li>Youtube</li>
</ul>
<p>Admittedly some of those sites (youtube, foursquare, twitter, yelp) would be supremely difficult to crawl and index, but it’s an interesting idea yeah?</p>
Auto rollback on error increase2013-11-26T00:00:00+00:00http://bnolan.github.com/2013/11/26/auto-rollback-on-error-increase<p>Last night I deployed to ZoomIn, and I forgot to run the migrations that were included in the deploy. Bit of a newbie mistake, but it did point out to me that I need better ops. There’s two things that would’ve helped in this case.</p>
<h1 id="auto-rollback-on-error">Auto rollback on error</h1>
<p>I use <a href="http://raygun.io/">raygun</a> for my error tracking, and they promptly emailed me after the deploy that the rate of errors had spiked majorly. They currently <a href="http://raygun.io/faq/does-raygun-have-a-public-rest-api">don’t have an API</a> though. Once they do have an API, I’d like to add some kind of job (cronjob?) that polls raygun for the error count, and if the errors/minute rate goes up after a deploy, automatically rollback to the previous version of the app.</p>
<h1 id="deploy-feature-branches-to-staging">Deploy feature branches to staging</h1>
<p>Another neat feature would be an easy way of deploying feature branches to staging, then when I wanted to push out the feature that I worked on, it wouldn’t have broken production.</p>
<h1 id="ops-scripts-for-small-companies">Ops scripts for small companies</h1>
<p>If I worked at a large company (10+ developers), I’d totally invest in getting these features built, but because it’s just me, it’s not really worth my time to create these scripts (there’s a lot of other ops stuff that needs to be done first, eg moving to s3, CDNs), but if I do get around to building these ops tools (probably as capistrano add-ins?), it’d be super cool to share them with the community.</p>
Mini rave for kiwiburn2013-11-25T00:00:00+00:00http://bnolan.github.com/2013/11/25/mini-rave-for-kiwiburn<p>I’ve been daydreaming about what kind of art piece I could take to <a href="http://kiwiburn.com/">kiwiburn</a>. Last year way my first kiwiburn so I didn’t take much apart from my lovely self and my acoustic guitar, but I did camp with Sam, who contributed towards a dance tent. What I’ve been thinking about taking, would be a motion activated mini rave. I’d grab the subwoofer and 4 channel amp out of the car, and hook it up so that two channels ran the sub, and the other two channels ran some home theatre speakers we have lying around.</p>
<p>I’d then hook up the whole setup to a 12 volt car battery and have the whole thing triggered off a motion sensor + arduino with a 12 volt relay. Then, hide the thing away in some dark corner of kiwiburn in the forest (towards the sound camp, but hopefully in some kind of nice grotto) and have a little PIR motion sensor that fires up the stereo, as well as a 12 volt led strobe light, for about 90 seconds, cranking out some Knife Party, or Nero or Far Too Loud. And then keep the music going for as long as people are still partying, up to a maximum of 5 minutes, then turn off so that people can wander away and then someone else can wander up and find the surprise disco.</p>
<p>I like this idea because it doesn’t involve towing a whole trailers worth of gear to the venue, doesn’t cost too much (the biggest expense is the sub and amp, and I could just chain them up to a tree), and might be a nice surprise for people exploring through the forest.</p>
Social media aggregator2013-11-21T00:00:00+00:00http://bnolan.github.com/2013/11/21/social-media-aggregator<p>After a brief google, I couldn’t find any sites that do this - which I find really surprising, but someone needs to build a tool that lets you create a ‘channel’, which is an aggregation of a bunch of Facebook pages and Twitter feeds into one easy to read page. It’s like a google reader, but just for facebook and twitter pages.</p>
<p>The use case for this is because I don’t want to be friends with all the bars and restaurants I often go to, but I would like to be able to go to a page and see what they’re all up to. Currently I do this by visiting their individual pages and seeing what the haps is. You can use Twitter lists for some of this, but about half the bars (Goldings for example) update their Facebook pages way more often than they update their Twitter feed.</p>
<p>My initial thoughts around this idea were based on the app <a href="http://vimeo.com/49793255">Spindle</a>, which was formed by some ex-microsofties and ran for about 2 years before they realised they weren’t going to get any traction and did an acquihire to Twitter. So, my initial thought was to replicate what they did, and hopefully hit the market at the right time, which I did as a prototype at venues.pw, but after a few weeks I used it less and less, but upon doing a bit more thinking, I think you could totally make a service that showed you all the twitter and facebook feeds in your city on a map, also maybe a Pinterest-style <a href="http://masonry.desandro.com/">jQuery masonry</a> display, since most of the restaurants and cafes regularly post images of their specials or new dishes.</p>
<p>Anyway, that might be my Monday mission, to build out a better prototype of the ‘social media aggregator’ for your city. It’d be great to go to one page and be able to see what’s going on at Baobab, Death Ray, Monterey, Wayward, Bebemos and Peoples Coffee.</p>
<p>Maybe <code class="highlighter-rouge">aggregator.com/feeds/newtown</code>?</p>
3d printing rockets2013-11-20T00:00:00+00:00http://bnolan.github.com/2013/11/20/3d-printing-rockets<p>Back in 2006, when I was first working on ZoomIn, I decided I needed a hobby aside from computer programming (startups being quite a full on project). I had been reading the <a href="http://armadilloaerospace.com/n.x/Armadillo/Home">Armadillo Aerospace</a> blog, I decided that I wanted to try and build a liquid rocket engine. So I bought a combination metal lathe and mill and $200 of aluminum stock.</p>
<p>I learnt a lot about metal work, how to cut the bar stock with an angle grinder, how to finish the cuts to a nice clean 90’ angle. How to center on the lathe, how to use a compound slide, how to bore out a hole. I’ve still got a collection of lathe drill bits and a nice boring bar somewhere, along with digital calipers and assorted other bits and pieces.</p>
<p>So now that I’m moving into a new place with a good sized man shed (and electricity!), I immediately started looking for another lathe on <a href="http://www.trademe.co.nz/">Trademe</a>. But after a brief think, I decided that while milling another liquid fueled rocket motor out of stainless steel would be a fun way to spend winter evenings, I’d been there done that, I’d rather move onto the next stage.</p>
<p>So after a bit of googling I found the <a href="http://rocketmoonlighting.blogspot.co.nz/">Rocket Moonlighting</a> blog, where a guy has 3d-printed a rocket engine using Direct-Metal-Laser-Sintering. It’s an expensive process. Luckily his first motor came out right, because it cost several thousand dollars (more than the cost of the whole lathe plus all the metal I used to make my prototype engine), but the cool thing is that once you have your CAD file setup parametrically, you can just change a variable, email a design to shapeways, pay your money and wait a week and you get your 3d-printed motor in whatever size you want.</p>
<p>I’m probably not going to jump right into the deep end and start 3d printing and hot firing rocket motors in my back yard (I don’t have the disposable income for starters), but it did make me wonder whether I could 3d print a water powered rocket nozzle. That way I could work on most of the fun bits, ie valves, control systems, structure, and then just power it using a carbon-fibre paintball co2 cannister that forced water through a nozzle and hovered on its tail.</p>
<p>That’d probably be something that I could test in the back yard too - I’m not sure how cool the neighbours would be with me hot-firing a nitrous-propane engine on our lawn. I’ll definitely have a think about it, a 3d printed water-powered throttleable (solenoid PWM maybe?) rocket would be pretty awesome.</p>
Tesseract and Postgres Trigrams2013-11-11T00:00:00+00:00http://bnolan.github.com/2013/11/11/tesseract-and-postgres-trigrams<p>I was chatting with <a href="https://twitter.com/nicobrevin">@nicobrevin</a> at the pub the other week, and we were discussing idea for interesting side projects, and we went back and forth over email for a few weeks, until we came across the idea of building some kind of tool around scanning and OCRing your shopping receipts.</p>
<p>Nick thought we could do something around analysis of how you spend your money (a histogram of how much you spend on beer over the course of a year?) and I thought we could maybe do something to make expense reporting easier (which is a bit of a silly idea really, since there are about a dozen xero-add-ins that already do receipt scanning via an app), so although I couldn’t see a clear way to make money from receipt-OCR-ing, it did seem like a fun project, and there was some cool technology to play around with.</p>
<h1 id="tesseract">Tesseract</h1>
<p><a href="http://code.google.com/p/tesseract-ocr/">Tesseract</a> is ‘probably the most accurate open source OCR engine available’. I’ve been using it in a very simple mode, where I take a photo with my nexus 4 (not the worlds best camera), run the image through autolevels using imagemagick, then send it through tesseract. The accuracy isn’t amazing, but it’s pretty good given the circumstances. Out of the box, the accuracy is good enough to pull prices out of the receipt. Nick has been doing a bit of work on training the software to recognize supermarket receipts directly, and the accuracy he’s got (out of flatbed scanned receipts admittedly) is really impressive, close to 99% recognition.</p>
<h1 id="postgres-trigrams">Postgres Trigrams</h1>
<p>One of the things I wanted to do with the app, was the ability to scan all your receipts you get for warranty purposes, and be able to scan back over time with them, basically do a fulltext search on your purchases, so that I could easily find the receipt for that macbook case or whatever. The only problem is, with the OCR results I was getting from the nexus’s camera, you couldn’t really do a fulltext search since the line for a Tuatara Pilsner might look like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>TUOTARA 6xPack $18.99
</code></pre>
</div>
<p>Happily, I had been reading about the <a href="http://www.postgresql.org/docs/9.1/static/pgtrgm.html">Postgres Trigram extension</a>, which even has a mode that uses GIST indexes to speed up similarity queries. You can now do something like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>select * from receipts where similarity(content, 'tuatara') > 0.3
</code></pre>
</div>
<p>And that lets you do a index-scan fuzzy search of all your receipts. I thought this was pretty cool, by combining <code class="highlighter-rouge">similarity</code> queries, with Nicks improved OCR rules, you would have a searchable list of all your indexes.</p>
<h1 id="writing-an-android-app">Writing an Android app</h1>
<p>Currently I’ve got a rails webapp that uses html5 to capture images from my Nexus’s camera, and upload them to the server for Tesseract. However, I’d like to do some processing (autolevels, greyscale, resample) on the mobile before I upload (raw uploads are 4mb, which is pretty slow over 3g, and I think sub 400k uploads should be possible), so this seems like a good opportunity to sharpen up my Android chaps with an Android app.</p>
<h1 id="foursquare-integration">Foursquare integration?</h1>
<p>If I knock out an app, I was thinking I might use foursquare to suggest nearby venues when you upload a receipt, then you’d have receipts tagged to the place you bought your goods</p>
<h1 id="what-do-you-think">What do you think?</h1>
<p>Can you see any weight in this idea? Might be worth expanding upon - what would you do if you had a fully searchable list of all your purchases over the last year? We also reached out to the guys at <a href="http://www.paperlessreceipt.com/">Paperkut</a>, since they’re doing something really interesting about getting your receipts emailed to you, which would obviously cut out the middle man of having to scan in your receipts.</p>
Improving autocomplete on zoomin2013-11-01T00:00:00+00:00http://bnolan.github.com/2013/11/01/improving-autocomplete-on-zoomin<p>Just sitting here on a Friday thinking about how I could improve the autocomplete on ZoomIn. The problem is, when you search for <em>rogue and vag</em> and it doesn’t autocomplete to “the rogue and vagabond”. Also, if you type in <em>k sing</em> it doesn’t complete to ‘k.sing’.</p>
<p>I should create an autocomplete column, that is all the works in the name, with the stopwords stripped out, and all the punctuation, and the letters concatenated together. So the Vag would be <code class="highlighter-rouge">roguevagabond</code> and k sing would be <code class="highlighter-rouge">ksing</code>. Then it’d autocomplete correctly.</p>
<p>Also, I need to come up with some kind of ‘find near here’ search. I wanted to find the nearest vietnamese to Newtown, and there’s no way of doing that in ZoomIn. You can find vietnamese <em>in</em> Newtown <code class="highlighter-rouge">/wellington/newtown/tags/vietnamese</code>, but I need to add a query like <code class="highlighter-rouge">/wellington/newtown/nearby/vietnamese</code>. Also, fulltext search is way better than tag search, I need to get proper full text searching across all columns and comments working properly.</p>
<p>Maybe a task for monday.</p>
Radar: Remaking Dopplr in an afternoon2013-10-21T00:00:00+00:00http://bnolan.github.com/2013/10/21/remaking-dopplr-in-an-afternoon<p>So, I read last week that <a href="http://dopplr.com/">Dopplr</a>, the social travel darling, was being shut down by Nokia at the end of the month. This isn’t too surprising, since there has been no development on Dopplr since it was acquired by Nokia in 2009. But it is a bit of a shame, since Dopplr was a cool app, and was a great way to keep track of your trips and get lots of interesting stats and maps. So, I decided to remake Dopplr, and let people migrate their data off Dopplr and onto my new service, <em><a href="http://getradar.in">RADAR</a></em>.</p>
<h1 id="data-model">Data model</h1>
<p>I made a few decisions early on. Firstly, I was going to require twitter authentication, and use the twitter friends relationship to define friendships on Radar, I didn’t want to maintain my own <code class="highlighter-rouge">friends</code> model. I also decided that I wouldn’t import tips from Dopplr, since tips is done exceedingly well by Foursquare, and there’s no need to reinvent the wheel. So, on Sunday afternoon, I started knocking up a rails + postgres + google maps site to replicate the trips functionality of Dopplr.</p>
<p>Each trip costs of a trip model, with multiple legs. Each leg is a different city, and a trip start and returns in your start city. I use the google places autocompleter to get the location of leg of the trip, and use rgeo to calculate the length of your trip.</p>
<h1 id="using-a-postgres-array-for-friends">Using a postgres array for friends</h1>
<p>When a user signs in using twitter oauth, I get their <code class="highlighter-rouge">twitter_id</code>, and a list of <code class="highlighter-rouge">friend_ids</code>. I store the friend_ids in a <a href="http://www.postgresql.org/docs/9.1/static/arrays.html">postgres array</a> of bigints, which means that I can do a query like this to find out everyone on Radar who you follow, and who also follows you. I call these friends.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def friends
User.where("uid in (#{friend_ids.join(',')}) and #{uid} = ANY(friend_ids)")
end
</code></pre>
</div>
<p>This is a super easy way to keep track of who you are friends with, without building any UI or relationships myself. I have a cronjob that refreshes your friend_ids every once in a while, so if you follow someone on Twitter, it adds them to your friends list.</p>
<h1 id="modern-geospatial-tools">Modern geospatial tools</h1>
<p>When Dopplr was created in 2007, there was a dearth of good geospatial tools. For example, Matt and co had to import the geonames database so that hey could suggest City names when you entered a new trip. Luckily, there’s now tools like the google <a href="https://developers.google.com/places/documentation/autocomplete">places autocompleter</a>, so you can just rely on Google to suggest place names and return a latitude and longitude. I then store the latitude and longitude in a postgis <code class="highlighter-rouge">geometry</code> column. This means I can use <a href="https://github.com/dazuma/rgeo">rgeo</a> to calculate the distance of trips using it’s built in haversine formula.</p>
<p><img src="/images/radar-autocomplete.png" /></p>
<h1 id="dopplr-data-import">Dopplr data import</h1>
<p>Handily, Dopplr is offering a data export option, which lets you download a .zip containing some json of your trips. I used the ruby <code class="highlighter-rouge">zip</code> gem to open up your .zip file and import your existing trips. So far about 20 users have imported over 300 trips into Radar, which means we have a decent amount of data to start working on a raumzeitgeist for Radar.</p>
<h1 id="what-next">What next?</h1>
<p>Well, I’ve emailed a few news sites to see if they’re interested in covering a Dopplr remake, and I posted the site to /r/entrepreneur on reddit. I’ve also tweeted at everyone who expressed sadness about Dopplr shutting down, and asked @mattb if he’d be interested in tweeting to his followers about Radar. I haven’t had a heap of luck yet, but we’ll see how the next week or two goes.</p>
<p>If enough people sign up to make it worthwhile, I’d love to keep building out Radar, there’s a bunch of great Dopplr features that I haven’t replicated yet, but it really depends whether or not there’s still the demand love for Dopplr out there in the community, or has everyone moved on - and Foursquare / TripIt / Kayak provide enough of the functionality that a Dopplr remake is a bit of a dead end.</p>
<p>It’ll be interesting to see what happens.</p>
Zeptomap - Maps for the Mission District2013-10-14T00:00:00+00:00http://bnolan.github.com/2013/10/14/zoomout---maps-for-the-mission<p>So today I woke up and decided that I’d have a crack at seeing whether I can build something like ZoomIn, but for the Mission District of San Francisco. And whether it could be built using OpenStreetMap data. Presenting… <a href="http://www.zeptomap.com/">Zeptomap</a>.</p>
<h1 id="importing-data">Importing data</h1>
<p>I started by getting a boundary polygon of the Mission, and using it to make an .osm extract of the california open street map data. I used <code class="highlighter-rouge">osmosis</code> for this, and it took about 10 minutes to extract out all the data for the mission.</p>
<p>I ended up with 102 unique streets that cover all of the mission. I then wrote an importer that merged the street segments together (since there are multiple OSM <code class="highlighter-rouge">way</code> records per street). I did this by using <code class="highlighter-rouge">st_collect</code> and <code class="highlighter-rouge">st_linemerge</code>. Which turns out to be a bit of a mistake - as you can <a href="http://www.zeptomap.com/california/san-francisco/the-mission/15th-street/and/guerrero-street">see here</a>, where the line data has lots of jumps. I’m not sure how to fix this, but I designed my importer and URL scheme so that I can just reprocess the data at any stage as I come up with better ways of writing the importers.</p>
<h1 id="knocking-up-some-rails">Knocking up some rails</h1>
<p>I created a quick rails app using <code class="highlighter-rouge">activerecord-postgis-adapter</code> to connect to the postgis database. I created url schemes like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>'california/san-francisco/the-mission/:street/and/:cross_street'
'california/san-francisco/the-mission/:street'
</code></pre>
</div>
<p>To create nice guessable, composable and shareable urls to cover all the streets of the mission, and the 900 intersections between streets in the mission. Each intersection shows a streetview of the cross street, and links to the nearest intersections (bit flaky at the moment, but I’m working on it).</p>
<p>The street view shows the street with a polyline rendered where the street is, and a list of the cross streets.</p>
<h1 id="disqus-with-ease">Disqus with ease</h1>
<p>I initially was going to use <code class="highlighter-rouge">devise</code> and create a <code class="highlighter-rouge">Comment</code> model in my app, but it turned out to be much easier just to use <a href="http://www.disqus.com/">Disqus</a>. Especially once I found out that you can pull disqus comments out of their API. I could then easily insert the comments into my own database for doing geospatial queries, but without me having to create and maintain my own user database.</p>
<h1 id="linode">Linode</h1>
<p>I was about to deploy all this on <a href="http://www.heroku.com/">heroku</a>, but after the hassle of the last app I did on heroku, plus the fact that you have to pay $50 a month to use postgis, I decided to just reboot my 1GB linode and reinstall apache + passenger. I don’t have puppet scripts for this yet, but I’m talking to a friend this week about how to go about setting up some scripts for firing up VMs. It took about an hour to get apache, passenger and postgis all setup and the app deployed.</p>
<h1 id="sitemaps-and-google">Sitemaps and Google</h1>
<p>Part of this is an experiment to see whether google will index a site that provides a place to discuss all the streets and cross streets and intersections of San Francisco, so I submitted a sitemap, registered the domain with google webmaster tools, and we’ll see if the almight Googs decides whether or not to crawl the site.</p>
<p>I still think I can make an interesting site even if the Googs doesn’t index the site, but it would be pretty cool to be on the first page of results for <a href="http://www.zeptomap.com/california/san-francisco/the-mission/15th-street/and/utah-street">15th and Utah</a>.</p>
<p>In the longer term, I’d like this to become something like the sadly missed Everyblock. I’ll get started with a nice url scheme, lots of nodes for people to talk about, get searching working well, aggregate content together into easily discoverable views, and then see what happens.</p>
Result.success in node.js2013-10-11T00:00:00+00:00http://bnolan.github.com/2013/10/11/result.success<p>Last night I started hacking a chat server using node.js, redis and websockets. It’s a bit of a right of passage for node developers, but I’m actually working towards building an android app I’m tentatively calling riskyclicks. Basically, it’s a public webbrowser, where you can see what pages everyone else in the app is viewing, and if you and another user are on the same domain at the same time, you can chat with them. I thought it up as a nice way to talk to people reading about WWII on wikipedia, or people reading stuff, or people reading about space.</p>
<p>I decided to do it as an android app, because desktop-browser-extension versions of this idea have been done approximately a hundred times, and it never sticks. So I’m gonna learn me some node.js, some redis, some websockets and some android development. (I’m using websockets because it delivers messages all in one, instead of splitting your json packages over multiple packets). It’s going to be totally a rough prototype-app, since I expect no one will actually want to share their browsing history, and I don’t want to spend more than a week or two on the idea.</p>
<h1 id="callbacks-and-nodejs">Callbacks and node.js</h1>
<p>Okay, so I started writing my <code class="highlighter-rouge">Server</code> class in node.js, and pretty quickly came up against this code:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if msg.action == 'login'
if @login msg.name, msg.password
send { action : 'login', success : true }
# do something here...
else
send { action : 'disconnect', reason : 'bad login credentails' }
ws.close()
</code></pre>
</div>
<p>So if the server recieves a <code class="highlighter-rouge">login</code> action from the client, check the credentials and then log the user in. Except, as soon as I went to implement the <code class="highlighter-rouge">Server#login</code> method, I realised I had to query redis, and that wasn’t synchronous, so I couldn’t return true or false from the login method. Instead, I had to use a callback.</p>
<p>This is where I developed something nice I call Success/Failure/Exception. By adding a callback as the last parameter to my login function, I can return the result of the login to the block, and using coffeescript, you can use the nice <code class="highlighter-rouge">?</code> operator to write code that looks like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if msg.action == 'login'
@login msg.name, msg.password, (result) =>
if result.success?
send { action : 'login', success : true }
# do something here...
else
send { action : 'disconnect', reason : 'bad login credentails' }
ws.close()
</code></pre>
</div>
<p>The login method looks like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>login: (username, password, callback) ->
if typeof(username)!='string' or username.length < 3 or username.length > 20 or !username.match /^[a-z0-9]+$/
return callback Failure "Bad username"
key = "user:#{username}"
@redis.get key, (err, reply) ->
if reply && JSON.parse(reply).password == password
callback Success
else
callback Failure("Unable to find user")
</code></pre>
</div>
<p>I think this reads quite nicely. You call the callback, and pass it Success or Failure (optionally with a message). I also define Success, Failure and Exception like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Success = (message) ->
@message = message
@success = true
@
Success.success = true
Failure = (message) ->
@message = message
@failure = true
@
Failure.failure = true
Exception = (message) ->
@message = message
@exception = true
@
Exception.exception = true
</code></pre>
</div>
<p>This way, you can either do <code class="highlighter-rouge">callback(Success)</code> or <code class="highlighter-rouge">callback(Success("some message"))</code>. It gets a bit gross when you <code class="highlighter-rouge">return callback Failure</code>, but apart from that, I think it reads quite nicely, and lets you have success, failure and exception cases returned from your async methods. Next up I’m thinking of adding a production exception logger in the definition of <code class="highlighter-rouge">Exception</code>, with a stack trace.</p>
Google maps doesnt play nice2013-10-03T00:00:00+00:00http://bnolan.github.com/2013/10/03/google-maps-doesnt-play-nice<p>I find it weird that Google, the company that relies on the internet having nicely-spiderable, indexable and machine-readable versions of webpages (otherwise their search engine would be useless), seems to have no intention of creating nice indexable, searchable and shareable urls from it’s own web properties.</p>
<p>I’m talking about Google Maps of course. I was on the <a href="http://www.reddit.com/r/melbourne">melbourne reddit</a>, and someone posted a link to a google maps page. So the reddit spider reached out and tried to get an image that it could use as a thumbnail for the link, and this was the result:</p>
<p><img src="/images/google-maps-thumb.png" />
<cite>Screenshot of the melbourne reddit</cite></p>
<p>The reason that the reddit url fetcher got it so wrong is because the html for that link is <a href="view-source:https://maps.google.com.au/maps?q=Holland+Ct,+Flemington+VIC&hl=en&ll=-37.789213,144.937661&spn=0.001147,0.002411&sll=-37.860283,145.079616&sspn=1.173152,2.469177&oq=holl&t=h&hnear=Holland+Ct,+Flemington+Victoria+3031&z=19&layer=c&cbll=-37.789213,144.937661&panoid=u_51kfCmxoEWxWkiV1-ABA&cbp=12,291.85,,2,-8.9">awful</a>.</p>
<p>It’s full of embedded css and javascript, for performance reasons I guess. It doesn’t have any opengraph information with latitude / longitudes. It doesn’t have any readable html tags with useful content in them. Things like an <code class="highlighter-rouge">h1</code>, maybe an <code class="highlighter-rouge">address</code> tag. Maybe you could find some embedded vcard information using the vcard microformat markup?</p>
<p>Nope, nothing like that. And look at the url too:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>https://maps.google.com.au/maps?q=Holland+Ct,+Flemington+VIC&hl=en&ll=-37.789213,144.937661&spn=0.001147,0.002411&sll=-37.860283,145.079616&sspn=1.173152,2.469177&oq=holl&t=h&hnear=Holland+Ct,+Flemington+Victoria+3031&z=19&layer=c&cbll=-37.789213,144.937661&panoid=u_51kfCmxoEWxWkiV1-ABA&cbp=12,291.85,,2,-8.9
</code></pre>
</div>
<p>It’s awful. I’ve already posted about this, about how it’d be nice that if when you clicked ‘get map url’ it returned a nice url, something like:</p>
<p>http://maps.google.com.au/australia/vic/melbourne/holland-ct?view=satellite</p>
<h1 id="why-should-they-interoperate">Why should they interoperate?</h1>
<p>I had a chat with the guys at Google sydney back in 2006 about making nice interoperable URLs for Google maps, and they just shook their heads and said “Why should we make spiderable urls, if we want to be in the google results, we just tell the search team to include us in the search results”. Which explains their whole attitude. They don’t want google maps indexed, and fair enough. But they could work harder to make their urls and html be useful to other computers on the internet. Opengraph tags would mean nice thumbnails for google urls. Canonicalisation of URLs would mean that you could find all the pages on the internet linking to the same google map. A nice to read url would mean you could paste it into an email without having to use a link shortener. Google place pages would show up in google search results, and on sites that embed google search results.</p>
<p>It’d also mean they’d play nice with the rest of the internet community, a community that enables their search engine, and would probably love to embrace nice google maps urls.</p>
Heroku, VPS or AWS2013-09-30T00:00:00+00:00http://bnolan.github.com/2013/09/30/heroku,-vps-or-aws<p>So last night I started hacking on a realtime data analysis tool for ZoomIn. Basically, I’m inserting a bit of javacript in the bottom of ZoomIn pages, and that javascript digs out the opengraph tags and sends the latitude/longitude of the current page off to my data analysis tool. I’m doing it so I can generate heatmaps of activity on ZoomIn, and maybe work out whether it’s trying to run my own ads on the site.</p>
<h1 id="hosting-a-rails-app">Hosting a rails app</h1>
<p>I got most of the prototype written last night, and I decided to get the production infrastructure set up today so that I could run it in a dark mode, where my tool doesn’t make any change to the ZoomIn site, but I can test that my infrastructure can handle the traffic. I don’t want to host this on the <a href="http://rimuhosting.com/">rimuhosting</a> VPS that hosts zoomin, since it’s already pretty pegged performance wise, and I want to get some experiment with other boxes.</p>
<h1 id="linode">Linode</h1>
<p>First I tried setting things up on my 512mb Linode. But a few things went wrong (upgrading passenger to ruby 2.0.0 segfaulted apache spectacularly), and the latency from nz to linode cali is about 170ms, which is too much for this case.</p>
<h1 id="digitalocean">Digitalocean</h1>
<p>I was pretty tempted to try DO, since they have SSD VPS’s for very reasonable prices, but I didn’t go down this path because [a] latency is high from nz [b] worried about over-tenanted boxes (just read a few posts about people running miners on their hardware).</p>
<h1 id="dedicated-box">Dedicated box</h1>
<p>This project doesn’t warrant a dedicated box yet, but I would love to get one at some stage. Either build a 32gb ram + SSD box, or lease one from somewhere in the states.</p>
<h1 id="heroku">Heroku</h1>
<p>So I went back to deploying on Heroku, except you have to pay extra for Postgis on postgres databases, and I couldn’t work out how to get activerecord-postgis to work with the heroku deploy recipes. Plus my last heroku project fucked out because I couldn’t work out how to get the asset packager working properly. Plus, it’s not in NZ, and there’s no Australian availability zone for Heroku.</p>
<h1 id="ec2-sydney">EC2 Sydney</h1>
<p>I’ve wanted to try out AWS every since they released their sydney availability zone, so after a bit of mucking around, I managed to get my box up and running on ubuntu 12.04LTS on the sydney zone. The latency is great, less than 20ms from my house in New Zealand. Installing ubuntu, rvm, ruby 2.0, passenger, apache, postgres and postgis went perfectly, and everything is humming away now.</p>
<p>Plus, another advantage with ec2, is that if I do more with this project and need to scale it up, I can start to use all those automated tools for firing up and imaging ec2 instances from the command line. For now I’m just using the web tools though, and capistrano for deployment.</p>
<p>Next research will be whether postgres has the insert capacity to record all the visits to ZoomIn, and whether I can tune postgres or do I need to do move to something else like mongo or redis? Mongo has nice geospatial extensions of course, but redis doesn’t have anything built in.</p>
Optimizing for mobile2013-09-27T00:00:00+00:00http://bnolan.github.com/zoomin/2013/09/27/optimizing-for-mobile<p>I did some work earlier this week on optimizing the mobile view of ZoomIn. The old one was pretty broken, it tried to layout a desktop-style display on your mobile phone and looked pretty awful. The google ads didn’t work well either.</p>
<h1 id="single-column-layout">Single column layout</h1>
<p>I initially had hoped that I could use a responsive design to make a single set of <code class="highlighter-rouge">.erb</code> templates that rendered everything, but that quickly turned out too hard. For starters I’m using an older version of bootstrap, and I didn’t feel like upgrading the entire site to the new responsive bootstrap.</p>
<p>Secondly, the authentication system will take a bit of fiddling around to get working on mobile, so currently the mobile site is read only. That’s something I want to fix, since it’d be great to be able to review and comment on places while you’re out and about on mobile.</p>
<h1 id="so-duplicate-the-views">So duplicate the views</h1>
<p>So I duplicated the views, and removed the google maps code, it now uses google static map apis. I also came up with a totally seperate mobile css which works pretty well. There’s still a chance I’ll try using responsive layout for some of the pages, because it’d be great to be able to access the recent comments / recently popular on your mobile device.</p>
<h1 id="make-the-mobile-site-richer">Make the mobile site richer</h1>
<p>It was interesting working on the duplicated mobile views, because I added a few helpers that I should have had for a while - for example a <code class="highlighter-rouge">breadcrumbs</code> helper that takes a place and generates all the parent places in a breadcrumb list. It also helped me whittle down exactly what zoomin was for, and made me think of ways I could optimize the community to work better on mobile.</p>
<h1 id="still-the-community">Still the community</h1>
<p>I’ve still had no luck jump starting the zoomin community. I think I need to make sure the site is working well, that there is a return on investment in adding comments (for example a great looking user page, or maybe some kind of gamification), then do some outreach to try and get back to a 2006-era zoomin community engagement. Unless that ship has sailed and everyone comments on Facebook, Yelp and Localist now. We’ll see.</p>
The Possibility of a Van2013-09-24T00:00:00+00:00http://bnolan.github.com/2013/09/24/the-possibility-of-a-van<p>This isn’t a sure thing, but I’ve been thinking about buying a van to replace my little mx5 that I use to commute to work. It’s a pretty drastic change, and considering that 90% of my use of my car is to commute down the motorway, it seems like an odd choice, to forgo top down cruising for driving around in a box on wheels. But here’s my thoughts of why a van might be cool</p>
<h1 id="taking-stuff-with-me">Taking stuff with me</h1>
<p>The boot in my mx5 is tiny, and it has trouble holding all my stuff. Currently all I have in the boot of my car is some art gear (I haven’t beeen to art in over a year), and some togs for swimming in. If I had a van (with a couple of cupboards in the back), I could keep my wetsuit (for sunny days), my rock climbing gear, my art gear and maybe a surfboard in the back, ready for whatever adventure summer provides for me. It also means I’d be able to go snowboarding without borrowing Rissas car.</p>
<h1 id="cups-of-tea">Cups of tea</h1>
<p>If I get a van, I’ll probably buy another little gas cooker, kettle and install a big water jug in it somewhere. Some of my favourite times when we used to have a van, was parking up on the south coast, making a cuppa tea, listening to the stereo and lounging about in the sun sitting on some cushions.</p>
<h1 id="doof-doof">Doof doof</h1>
<p>There’s no roof in an mx5 for a subwoofer, and I’ve got a nice 12” one kicking around at home. If I get a van, I can install a decent head unit (one with bluetooth a2dp so I can stream spotify from my phone), some rocking 6.5” components in the door, and a sub in the boot.</p>
<h1 id="a-bed">A bed</h1>
<p>I’ve been looking at <a href="http://www.steamydave.co.uk/campervan/">tiny campers</a> on the internet, and I’ve seen some pretty <a href="http://titimus.co.uk/suzuki-carry-camper">cool campers</a> made out of vans as small as a suzuki swift. I’d probably get a van a little bit bigger than that, but I still love the idea of a campervan with a bed and a tiny kitchen in it. It’d make it so much easier to decide to go up to Kaitoke for the weekend, everything would already be in the van, no need to dig out all the camping gear from the spare room. Plus Rissa and I could also pack everything up and cruise off to the south island whenever we got 5 days off in a row.</p>
<h1 id="solar-power">Solar power?</h1>
<p>This is a bit of a stretch, but if I really got into my van, I could install a big giant battery from <a href="http://aasolar.co.nz/">aasolar</a>, a solar panel on the roof and a solar charger, then go off grid for the day. Imagine parking up on the south coast of Wellington, fire up the 3g hotspot, plug my macbook into an inverter and work for the day, making cups of tea and going for a midday swim.</p>
<p>So yeah, I haven’t decided whether to go the way of the van yet. There are a bunch of downsides, like it’s more expensive than the mx5, way less fun to drive around the Petone roundabout, probably use more gas, way less fun to drive on a sunny day, but it’s definitely something at the back of my mind.</p>
<p>Sorry for the non-technical post. Your normal technical transmission to resume shortly.</p>
Named Entity Recognition2013-09-18T00:00:00+00:00http://bnolan.github.com/2013/09/18/named-entity-recognition<p>I’ve been researching ways of gathering location data from forums like /r/wellington and /r/sydney. One of my experiments was to create a javascript tool that helped you create comments with embedded geo data (via links), another one was to try and convince a reddit to mark up all place names using *stars*. Both of these don’t work very well because they require a big shift in behaviour from users.</p>
<p>In the back of my mind I’d been thinking of ways of doing content extraction from the existing text corpus. You have a few advantages in doing this. Most of the content I want to import is targeted at a city, and once you have decent coverage of places in that city, you can look for those place names in text and start to match text to places.</p>
<p>I did a manual job of this, marking up a bunch of posts with their place names, and it was pretty slow going.</p>
<p>So last night, whilst enjoying a pleasant beer at the <a href="http://zoomin.co.nz/nz/lower+hutt/petone/jackson+street/146/-sprig+fern+tavern/">sprig and fern</a> in Petone, I was wondering whether it was possible to automatically extract place names from text. I’m not talking about the yahoo place finder code that recognized city names, I was wondering if it was possible to statistically analyze text and extract place names.</p>
<h1 id="ner">NER</h1>
<p>It turns out there is a bunch of research into this, and the best opensource tool is the <a href="http://nlp.stanford.edu/software/CRF-NER.shtml">Stanford Named Entity Recognizer</a>. So I wrote a script that crawls a subreddit, pulls out all the text from the comments, and ran it through the NER. It’s not great, but here’s the output for /r/sydney:</p>
<p><em>Sydney Tower Skywalk, Skywalk, Bridgeclimb, Skywalk, GTA, North Sydney, Broadway, Nah, PC, GTA, Oval, Ozgameshop, Ozgameshop, DSE, Congrats, Congrats, NSB, Breaking Bad, Factory Theatre, Macau, Redfern, Nice, Redfern, Newtown, Tupac, Kings, Williams, Good God, Ok, The Passage, DJ, Crunch, Parramatta, Olympic, Bondi, Hornsby, Millenium, Arena, Crossfit, Pennant Hills, Bondi, Crossfit, Bondi, Boxing, Greg Everetts, Catalyst, Coles, Surry Hills, Kodak Compact, First Google, Kent St, Try Trophy Land, Kent St. You, Pics, Mister Minit, ALDI Mobile, Telstra, Telstra, Australia, Australian, SIM, Vaya, Telstra, Optus WiFi, Virgin, Virgin, Optus, Telstra, Optus, Telstra, Optus, Whirlpool, Australians, Sydney, Best, Surry, Japworld, Princes Highway in St Peters, O’Brien, Happens….</em></p>
<p>And there’s a bunch more I stripped off.</p>
<p>Now, this isn’t that interesting, as there’s a lot of generic suburb names there, a bunch of peoples names and a bit of breaking bad. Actual references to places, I can only see Crossfit, The Passage, Ozgameshop and the Factory Theatre. But the thing is, is that you can train the NER, so it’d be possible to quickly go through the content, highlight place names, and retrain the classifier.</p>
<h1 id="place-names-are-in-uppercase">Place Names are in Uppercase</h1>
<p>You can also work off the fact that most place names are in uppercase, and you can also pre-seed your classifier with the names of all the streets in a city (using OpenStreetMap).</p>
<h1 id="why-is-this-interesting">Why is this interesting</h1>
<p>I don’t know how much analysis Google does on non-hyper-linked text, but I’ve not seen many search engine approaches that try and extract terms from plain text, and generate a link graph based on place names, instead of on hyperlinks. If you came up with a crawler that could explore a corpus of data, extract place names and generate links between the text and places. I’m not sure where you’d get with it, and if you’d end up with something much more interesting than just a full text search on the same corpus, but it’s an interesting idea.</p>
<p>You could easily plot a map of what everyone was talking about. And you could establish links between places, to say ‘you’re insterested in this, you may like that’. And all without anyone having to change their habits or use new technology.</p>
jQuery Mentions and the Places API2013-09-16T00:00:00+00:00http://bnolan.github.com/2013/09/16/jquery-mentions-and-the-places-api<p>So I started today with grand plans of doing some data-import work on ZoomIn, but sadly I got sidetracked. It’s a bit of a shame really, because ZoomIn needs more attention, and the traffic is starting to suffer because I’m not putting enough effort into it. Although, ironically, I’m making more money from adsense than ever before, since I’ve improved the site performance and got better at placing ad units.</p>
<p>But anyway, this morning I had an itchy idea that I felt like working on, one where you could use the Facebook-style @mentions functionality to write a comment about places. So for example, if someone asked me: “Where’s the best craft beer in Wellington”, I could reply something like this:</p>
<p>The best craft beer is at @Hashigo Zake, @Goldings Free Dive and @Little Beer Quarter.</p>
<p>So I did a bit of research and discovered the <a href="http://podio.github.io/jquery-mentions-input/">jQuery Mentions plugin</a>, which is a great bit of tech which lets you have a textfield that has an inline autocomplete, whenever you type @something. I did a bit of prototyping, and got the autocompleter powered by the Google Places autocomplete API.</p>
<p>So now you could totally write comments with links to the actual places, without having to google for each place and find the link and embed it in your content. It was a pretty cool proof-of-concept.</p>
<h1 id="and-thats-where-i-should-have-stopped">And that’s where I should have stopped</h1>
<p>I got the proof of concept going, and with a bit of playing, it was clear that it was a cool idea, but my hunch was that it wasn’t worth spending any more time on. But foolishly, I decided to dive on and see if I could make a tool out of this little one-trick-feature.</p>
<p>I do a lot of my project planning (on my own projects) based on hunches. I sit down in the morning, and unless I’ve already planned what I’m going to do, I start by following my nose, and soon enough I find I’m working on something useful and hopefully something I can knock out in one day.</p>
<p>The counterpoint to that, is when you’re working on something and it feels like you’re pushing shit uphill, you should drop what you’re doing, and write off the development you’ve done so far. Maybe commit the current branch, then flick back to master and work on something else.</p>
<h1 id="commentmap"><a href="http://www.commentmap.com/">Commentmap</a></h1>
<p>Instead, I decided I would take my proof-of-concept that let you mention places, and turn it into a tool for writing hyperlinked reddit, bbcode, or html comments. This took another 3-4 hours. I bought a domain name (seldom a worthwhile investment), set up heroku instance and deployed the thing. This meant that I didn’t get any ZoomIn work (especially importing all the wikipedia / OSM data I’ve got kicking around) done, but I did get my little experiment out there and on the internet.</p>
<p>If you go there, and enter a city, then try and write a comment about that city (maybe recommending a few places), and hit the @-key before you type a comment, you’ll see the autocompleter in action.</p>
<h1 id="maybe-worthwhile">Maybe worthwhile?</h1>
<p>While commentmap, in it’s current form, isn’t going to get any traffic or probably be useful to anyone but me, it has been useful for me to experiment with the idea. I’ve used it to write a post about my <a href="http://www.reddit.com/r/sydney/comments/1mhg90/my_experience_of_sydney/">experiences in sydney</a> on Reddit, and also to reply to some questions about <a href="http://www.reddit.com/r/Wellington/comments/1mhbd9/my_favourite_craft_beer_places/">craft beer in Wellington</a>. So, while I foolishly thought I might be building something people wanted to use, what I’ve really done is build something that I wanted to use.</p>
<p>And maybe if I find myself re-using this tool over and over, maybe over time it’ll twig some little idea in the back of my mind, that I can add to and extend, to eventually make this into a cool tool in its own right.</p>
<p>Maybe experiments that seem like a bad idea are worth doing just to prove they are or aren’t.</p>
Zoomin updates2013-09-15T00:00:00+00:00http://bnolan.github.com/zoomin/2013/09/15/zoomin-updates<p>I’ve been pushing a few new changes out for ZoomIn today. This is just a quick recap - but here’s what I’ve done.</p>
<h1 id="new-header">New header</h1>
<p>I redesigned the header to take up less space so that there’s more room for content on the page.</p>
<h1 id="redesign-region-pages">Redesign region pages</h1>
<p>I redesigned the region pages to have a nicer display of suburbs and streets in that region, a better ad block (the old one wasn’t performing well), and a tag cloud of most popular tags. I’ve also removed all the ‘dots’ off the map, since some people found them obnoxious.</p>
<h1 id="redesign-home-page">Redesign home page</h1>
<p>I redesigned the home page to load faster, have less content splatted in your face, and to link to the explore pages.</p>
<h1 id="fixed-up-explore-pages">Fixed up explore pages</h1>
<p>These are the recent comments, photos, places pages. I’ve updated them all so that they render correctly and look good on the site. You can now quickly navigate through and see recent members or other content.</p>
Mapping a city reddit2013-09-12T00:00:00+00:00http://bnolan.github.com/2013/09/12/mapping-a-city-reddit<p>I started working on this last night but didn’t get it to the point of making a prototype or screenshots, maybe tonight. Anyway, as part of my investigation into different models of geo-based messaging, I was wondering what it would look like if you mapped all posts on the <a href="http://reddit.com/r/wellington">Wellington subreddit</a>.</p>
<p>So I started by manually geotagging the first 50 posts in the reddit. I ended up with only about 20 geotaggable places. I think you could automate this if you made a rule on the reddit that whenever people mention places, they put stars around the place name. eg “I think *midnight espresso* is a great place for coffee”. Then I could write a relatively simple scraper that crawled the reddit for new posts, then looked up that place name on foursquare.</p>
<p>Anyway, after manually crawling those 50 posts, I discovered that people don’t actually mention specific places very often on the wellington subreddit. And then after looking at the sydney reddit, I saw a similair thing, only about 30% of posts have specific places mentioned in their comments.</p>
<p>This kind of blows my idea out of the water, I was thinking of creating something like the Facebook ‘tag people in this post’ functionality, a javascript extension that meant that you could reply to a post and type @something whenever you wanted to refer to a place, and it would autocomplete the place name using the google places API.</p>
<p>Quora has some city-based discussion nodes now. They don’t seem to have integrated with foursquare or the google places api, so that when you reply to a question and mention a place, there’s no link to the places. Anyway, I’ll keep experimenting and hopefully strike that combination of comments, maps and locations that is fun and easy to use and you could grow a community around.</p>
<p>I’ll update my blog when I’ve got the map of <code class="highlighter-rouge">/r/wellington</code> working.</p>
Postgres as a service2013-09-06T00:00:00+00:00http://bnolan.github.com/2013/09/06/postgres-as-a-service<p>I had this idea the other day while daydreaming about javascript single page apps. I thought it’d be cool to have postgres available as a json service via CORs, that you could build little javascript apps with a centralised datastore. Just write a small bit layer of ruby that takes a query, executes it on the postgres server and returns the results at json. Obviously you could only share the databases with people you trust, since anyone could run:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Backend.query("drop all from posts");
</code></pre>
</div>
<p>But, for example, inside on organisation, you could build little apps that add functionality onto bigger apps. Imagine greasemonkey scripts with a shared database. It’d be nice to integrate something like <code class="highlighter-rouge">phppgadmin</code> into the service, then eople could log in and see their databases and admin them.</p>
<h1 id="backbone-integration">Backbone integration</h1>
<p>It’d be worth writing a backbone integration that made Backbone.sync work with the database. I don’t think that’d be too hard, you could even come up with a nice model for creating migrations in Javascript.</p>
<h1 id="fork-databases">Fork databases</h1>
<p>It’d be great to be able to fork and backup databases whenever you want, like you can on Heroku, so that if you’re making dangerous changes to your app you can keep backups.</p>
<h1 id="pricing">Pricing</h1>
<p>You could probably give away small limited-size databases (with a 1000ms query time limit and 500 row limit), so that people could knock up little ideas on the service, and then have paid plans for bigger databases. You could also have some kind of nice referral-sign-up system, where someone creates a single-page-app that does X, and then they host the app themselves on S3 or Heroku, and when someone wants to use their app, you have an API that lets people sign up for a paid database through a javascript API, and then you pay the app developer some porportion of the monthly revenue for databases created by their app.</p>
<p>Anyway, it’s an interesting idea, and I might have a crack at writing the backbone sync module at some stage.</p>
Script defer2013-09-03T00:00:00+00:00http://bnolan.github.com/zoomin/2013/09/03/script-defer<p>So as part of my push to increase the traffic to Zoomin 4-fold (which is a crazy ambitious plan, but everyones gotta have a goal right?), I decided to try and make the site load as fast as possible. <a href="http://twitter.com/johnclegg/">John Clegg</a>, the former owner of ZoomIn, was always a fiend for making sites load fast, and a big proponent of using yslow, so ZoomIn was already pretty speedy, but after playing with it yesterday, I was sure I could make it faster.</p>
<h1 id="google-maps">Google maps</h1>
<p>The biggest slowdown on the site, as far as I could tell, was loading the google maps api and the massive javascript that powers ZoomIn. The google maps javascript is mostly-cached and minified, but the browser still takes time sending an <code class="highlighter-rouge">if-changed-since</code> request to make sure it has the newest version. There is no way around this slowness, and if the google maps api is in the <code class="highlighter-rouge">head</code> of your document, you’ve got another 100ms or so that you can’t code around.</p>
<h1 id="move-it-to-the-bottom">Move it to the bottom!</h1>
<p>So I grabbed all the external APIs I was using, and stuck them at the bottom of the body tag, leaving only the stylesheet (which is aggresively cached by apache) in the <code class="highlighter-rouge">head</code> section. However, ZoomIn has a lot of inline javascript that instantiates the different map styles and visualizations, and these all expected the Javascript to be included in the head. Easy fix, just wrap those in a block that gets run after everything else.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>function Ready(callback){
Ready.callbacks.push(callback);
}
Ready.callbacks = [];
</code></pre>
</div>
<p>Then your code looks like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Ready(function(){
new GoogleMapsInstance(some params);
});
</code></pre>
</div>
<p>And at the bottom of the page, after all the javascript has been loaded, I’ve got this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>while(func = Ready.callbacks.shift()){
func(jQuery);
}
</code></pre>
</div>
<h1 id="its-like-script-defer">It’s like script defer</h1>
<p>It would’ve been nice just to use the <code class="highlighter-rouge">script defer</code> attribute to make all the javascript run once the DOM has become interactive, but reading the <a href="http://caniuse.com/script-defer">caniuse page</a>, it says defer is only “partially supported”, and I really didn’t feel like debugging IE heisenbugs, so I rolled my own solution.</p>
<h1 id="how-does-it-feel-now">How does it feel now?</h1>
<p>Snappy as hell.</p>
Removing facebook2013-09-02T00:00:00+00:00http://bnolan.github.com/zoomin/2013/09/02/removing-facebook<p>Well today I discovered something awesome. The facebook cross-domain framework crashes in my install of IE7. And I’m guessing it does the same in an unknown number of client browsers out there. So I tried debugging it for a while. The problem happens in <code class="highlighter-rouge">fb.init(..)</code>, and I’m guess it’s because of the flash embed that Facebook uses for cross-domain communication.</p>
<p>I get an <code class="highlighter-rouge">error 153</code> exception, with no debugging information.</p>
<h1 id="shall-i-fix-facebook">Shall I fix Facebook?</h1>
<p>I added Facebook auth because I thought it would be faster and easier for people to sign up using Facebook than using the traditional username/password setup that Zoomin has. So I added facebook auth, and in the last 9 months - these are the sats:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>zoomin_nz=# select count(id) from users where created_at > '2013-01-01';
count
-------
933
(1 row)
zoomin_nz=# select count(id) from users where created_at > '2013-01-01' and uid is not null;
count
-------
165
(1 row)
</code></pre>
</div>
<p>What this means is that only 18% of signups are with Facebook accounts. This mirrors what someone commented on my blog last week, was that I should consider adding other authentication methods to ZoomIn, because not everyone wants to share their Facebook credentials with a site.</p>
<p>And I can’t count all the people who have tried to log in using Facebook, or even just access the site and their browser has crashed out due to the IE7 bug.</p>
<h1 id="rolling-back-facebook">Rolling back Facebook</h1>
<p>So it looks like, for now, I’m going to roll back the facebook login. I’ll send an email out to the 165 people who signed up via facebook and let them know that they can just go to the reset password link to change their password and login without Facebook auth.</p>
<h1 id="why-do-people-prefer-loginpassword">Why do people prefer login/password</h1>
<p>My guess is that a lot of the people who sign up to ZoomIn are signing up to edit their business details, and don’t want to have to use their personal facebook account to connect with ZoomIn.</p>
<h1 id="will-i-re-add-facebook-auth">Will I re-add Facebook auth?</h1>
<p>I’ll have a think about it, but for now it’s not on the list.</p>
<h1 id="fixing-login-bugs">Fixing login bugs</h1>
<p>I’ve got a few email lately about people having problems resetting their passwords or logging in. I’ve done a bunch of debugging, and fixed many skanky manky bugs in the login process, so hopefully people will have a lot more luck logging into ZoomIn. If you’re a ZoomIn user and can’t log in or reset your password, or signup - please let me know and I’ll look into it! :)</p>
Removing groups functionality2013-08-29T00:00:00+00:00http://bnolan.github.com/zoomin/2013/08/29/removing-groups-functionality<p>I’ve been working on ZoomIn a bit lately, trying to work out how I can grow the traffic from the current 10,000 a day, to something about twice that. It’s a huge ask, since most of the traffic to ZoomIn is just people finding it via Google search results, and once someone goes to the ZoomIn site and sees the map, they tend to leave right away (or click on a Google ad).</p>
<p>But it’s my best chance of building and deploying code that people will actually see, so I’ve been hacking on ZoomIn, trying to thing of what I can do to grow the traffic. One of the things I’ve been doing lately is removing functionality that people don’t use.</p>
<h1 id="groups">Groups</h1>
<p>When I first created ZoomIn, I was aiming to make a community site like Flickr, so one of the Flickr features we copied was Groups. The idea of a group is that people could create a little group, with a forum, and then add places to that group. A good example would be ‘mountain bike tracks’, which might have tracks added from all over New Zealand.</p>
<p>People liked the idea, and when we created groups back in 2006, quite a lot of groups were created.</p>
<p>But fast forward to 2013, and less than 0.5% of the page views on ZoomIn go to the groups pages. And so, in the interest of having more wood behind fewer arrows, I’m going to remove the groups code from ZoomIn. Note that I’m not actually deleting the group models from the database, all the existing groups will still be there, but I’m going to remove the controller and views, so that I can focus on making the common use-cases of ZoomIn work much better. If I get ZoomIn growing again, I’ll look at re-adding the Groups in a crazy-improved guise.</p>
<h1 id="login">Login</h1>
<p>Weirdly, the second most popular page on ZoomIn is the <code class="highlighter-rouge">/login</code> dialog, which makes me think that maybe there are more people trying to comment and use the social components than I expected, and that they are having trouble logging into the site (or else being turned off by the ‘you must login screen’ and abandoning their comments). I’ve been tentatively thinking about allowing anonymous comments on the site.</p>
<h1 id="improvements-to-region-pages">Improvements to region pages</h1>
<p>I’ve been working hard on the region page, redesigning and redesigning, trying to find a way to use the traffic to <code class="highlighter-rouge">/petone</code> and <code class="highlighter-rouge">/newtown</code> to create a community around those places. I’m a very long way away from creating the subreddit-for-each-town site that I’d like it to be, but I’m giving it a stab. I’ll keep you updated.</p>
Dispersion, a webRTC social network2013-08-28T00:00:00+00:00http://bnolan.github.com/2013/08/28/dispersion,-a-webrtc-social-network<p>So after discovering <a href="http://peerjs.com/">peer.js</a> and daydreaming about a peer to peer social network that looked like Facebook but worked like instant messaging, I had a bit of a hack at it, and it’s super early stages, but I’m calling it <a href="http://github.com/bnolan/dispersion/">dispersion</a> at the moment, since dispersion is listed as a synonym of Diaspora. ;)</p>
<p><img src="/images/dispersion.png" />
<cite>Screenshot of the prototype</cite></p>
<p>It’s probably not worth your time checking out the repo yet, it’s still so alpha and buggy that it won’t work for you in all likelihood. It’s developed using <a href="https://npmjs.org/package/capt">capt</a>, my javascript buildtool (which I really need to work on, but that’s a diffferent post), in coffeescript and backbone.js.</p>
<h1 id="localstorage">LocalStorage</h1>
<p>Your posts and your friends posts are all serialized to localStorage as json, so that the app can fire up instantly when you load it. I saw the <a href="http://crypto.stanford.edu/sjcl/">SJCL</a>, which would be a nice way to keep an encrypted copy of your localStorage up on the internet somewhere, so you could log into the social network from multiple locations.</p>
<h1 id="devise">Devise</h1>
<p>I was thinking of hacking up peer.js to have authenticated sessions, but after looking over the code and the issues on the peer.js github page, I think I’m misunderstanding what peer is for. So it looks like instead, I’ll create a simple app on heroku that uses devise for authentication, and provides a CORS service that lets you search for users by their full names, instead of just adding them by their handle. The heroku service would keep track of whatever your current peer.js identifier is, and tell your friends where to find you.</p>
<h1 id="prototype">Prototype</h1>
<p>Anyway, this is all just a super rough prototype at the moment, I’ll do a proper post when I’ve had some more time to work on it, which might be a while, since I’m pretty busy with the <a href="http://www.powershop.com/">day job</a> and ZoomIn. One thing I would like to add would be some sweet jasmine and cucumber tests, since peer to peer stuff is pretty complicated and it’s a pain in the arse to test manually.</p>
New Fonts and Lines2013-08-27T00:00:00+00:00http://bnolan.github.com/2013/08/27/new-fonts-and-lines<p>It’s awesome having a designer sitting downstairs with us. I asked <a href="http://twitter.com/kellective">Kelly</a> whether my blog still looked okay (since I designed it about 3 years ago). She said that I should get rid of the double lines and change the header fonts. One quick visit to google fonts later, and we’re using <a href="http://www.google.com/fonts/specimen/Open+Sans">Open Sans</a> for headings and all the dotted and doubled lines have been replaced with <code class="highlighter-rouge">1px solid #ccc</code>.</p>
<p>That’s all for the blog post today, bit busy with mobilification of various bits of the Powershop website. But I did do some work on that webRTC social network over the weekend, it kind of came together okay. Then I didn’t get anything done on Zoomin on monday, because even though I think it’s my best bet as for building something that will actually make an impact (10,000 people visit Zoomin on a monday, I’ve got a built-in audience), I still don’t know what exactly to build to make the social aspect of ZoomIn take off again.</p>
<p>Any ideas for the socialisation of <a href="http://www.zoomin.co.nz/">ZoomIn</a> are greatly appreciated.</p>
Social Network over WebRTC2013-08-22T00:00:00+00:00http://bnolan.github.com/2013/08/22/social-network-over-webrtc<p>This idea popped into my head while playing with the <a href="http://peerjs.com/">peer.js</a> demos, and as a partial response to all the GCSB / PRISM bullshit. You could build a pretty sweet little social network just using webRTC, without a central server.</p>
<p>I’m thinking of a facebook-wall style network, where each user has a wall, and you can befriend anyone else. You would first start by letting clients use a twitter-style handle as their node id, and adding authentication to the peer server, so that you can prove you own a particular handle. Note that nothing would be stored by the central server except a username and password (and whatever data webRTC needs you to hold to broker sessions), everything else would be stored in localStorage on a client.</p>
<h1 id="making-a-friend">Making a friend</h1>
<p>To make a friend, you would add them by their handle, and the broker server would tell you the IP address and port number (via STUN) of the client you want to connect to. That client then accepts you, and sends you any updates to their wall, and your client can post any updates to that persons wall.</p>
<h1 id="json">JSON</h1>
<p>Peer.js supports data channels, so you can send json over the webRTC connection. So you do all your user interface in Backbone.js, and serialize it to JSON to store in localStorage, or to send over webRTC></p>
<h1 id="only-while-online">Only while online</h1>
<p>The biggest problem with this setup of course, is that it only works when everyone is online at once. So, for example, it’ll work when everyone is at work during the day, or if you and a friend are both online in the evening. You can even post to your own wall when friends are offline, and if you wanted to, when a friend connects to your webRTC instance, they could poll you to say ‘send through all the updates to your wall since timestamp x. It’s a bit like instant messenger, you have to leave the app open in a browser window somewhere.</p>
<h1 id="no-encryption-required">No encryption required</h1>
<p>It’s kind of cool, because it doesn’t require any easy-to-fuck-up cryptography to provide a relatively private social network, since all the content is stored on peoples PCs and not in a cloud server. It also means you can scale the network to zillions of people relatively easily, all your server has to provide is STUN and authentication, all the content (including images and any other media you want to share) is stored in the browser and sent directly from peer-to-peer. The biggest problem I had with knocking up a distributed social network like Diaspora-X or working on Buddycloud, was that you had to build this fuck-off big federation component that was hard for people to install, and really hard to do properly. With a peer-to-peer solution, there are a bunch of downsides, but a massive upside in that scaling is free.</p>
Hipstergram2013-08-20T00:00:00+00:00http://bnolan.github.com/2013/08/20/wapstergram<p>I really wanted to buy a <a href="http://www.gsmarena.com/nokia_8910i-370.php">Nokia 8910i</a> back in the day. And I still kind of want to, I mean damn, magnesium case and a shiny silver keyboard, look at it, it’s a sexy phone.</p>
<p><img src="http://www.dubai-mobiles.net/images/phone_pics/rare/nokia_8910i.gif" /></p>
<p>So you get an oldschool phone like that (released in 2003) - if you can even buy it, since they go for about $300 for a good condition one on ebay. But what can you do with it? Well you can text people. You can call people. You can play snake. And you can surf the internet on the WAP-compatible browser. That raises the possibility of creating a unique old-phone-only social network. I know that you can surf facebook on wap-phones (that’s Facebooks plan for getting the next billion people on Facebook), but I’m talking about rolling out a ruby-on-rails-based (<code class="highlighter-rouge">has_many :friends</code>) social network that does some user-agent sniffing so that it’s only accessible to older feature phones.</p>
<p>My only concern is that the 8910i, my phone of choice, might be too old and only have <code class="highlighter-rouge">WAP</code>-instead of <code class="highlighter-rouge">xhtml-mp</code>. Writing a wap site would be kind of painful because it uses different markup for links and forms, so you couldn’t use all the built in rails helpers. But yeah, if I knew enough hipsters to get this thing off the ground, I’d totally invent Hipstergram, the social network for people with old phones.</p>
<p>Because surely there’s going to be an ironic backlash against modern do-it-all phones and a return to simpler phones that are only good for updating a text status, SMSing and calling people.</p>
Zoomin Work2013-08-19T00:00:00+00:00http://bnolan.github.com/2013/08/19/zoomin-work<p>It’s monday, so I’m working on Zoomin. I spent some of the weekend trying to add ‘topic pages’ to the wiki, so that you could create a page called ‘Bohemian places in Wellington’, and then write about the bohemians of wellington, revelling in their bohemia, and have that page link to all the relevant anarchistic squats, indie cafes and alternative pubs. That work isn’t quite ready for primetime yet though, so I’m leaving it on the backburner and working on tidying up a few things.</p>
<h1 id="removed-add-photo-link">Removed ‘add photo’ link</h1>
<p>Add photo doesn’t seem to be working properly at the moment, so I’m going to remove the ‘add photo’ link until I have time to make it work properly.</p>
<h1 id="restyle-the-region-pages">Restyle the ‘region’ pages</h1>
<p>These are the pages like <a href="http://www.zoomin.co.nz/wellington/">/wellington/</a>, which show details for a region. I have made the map after the page loads, for better performance and to make it more obvious there is content further down the page. I have also removed all the ‘dots’ that showed comments on the map. They kind of the map less useful in retrospect.</p>
<h1 id="working-on-wiki-pages">Working on ‘wiki’ pages</h1>
Facebook for Cities2013-08-16T00:00:00+00:00http://bnolan.github.com/2013/08/16/facebook-for-cities<p>I think the hyperlocal nut is really hard to crack. I found <a href="http://recoveringjournalist.typepad.com/recovering_journalist/2007/07/backfence-lesso.html">this article</a> by one of the founders of Backfence, an early attempt to build hyperlocal sites. There was also a follow saying how Backfence was beaten by the <a href="http://publishing2.com/2007/07/09/wrong-on-hyperlocal-google-and-web-10-killed-backfence/">existing ad-hoc forums</a> that had been around for years before Backfence tried to make a community site for cities and neighborhoods.</p>
<h1 id="nextdoor">Nextdoor</h1>
<p>Nextdoor is doing a really interesting job of cracking the hyperlocal nut, by making online communities small and with real identities, so people feel free to share their information on there. Sadly, it seems like the nextdoor experiment might be swaying to being a ‘security-social-network’, where people just post their concerns about strange cars in the neighborhood, or updates from the police. (This is is all anecdotal, from following their blog, since I can’t access any of the nextdoor neighborhoods for obvious reasons).</p>
<h1 id="rwellington"><a href="http://reddit.com/r/wellington">/r/wellington</a></h1>
<p>I think the city-reddits are a really great example of what a hyperlocal community could look like. I think if you combined something like this with the <a href="http://localwiki.org/">localwiki</a> effort, you’d have a pretty neat little bit of technology. The problem I see, is that it looks like if you worked with the city reddit mods and asked them what you could build to make their life easier, you could eventually come up with a feature set that was superior to the city-reddit experience, but how on earth would you cold-start the communities. You’d have to build something that people can’t do anywhere else on the internet, some unique feature that people would invite their friends to use the site.</p>
<h1 id="unique-feature">Unique Feature</h1>
<p>For Facebook it was the ability to upload information about yourself so that people could find you (remember this was back in 2004, and that was a unique proposition). Instagram has filters. The cold start problem really if the hard part. I think a lot of people can imagine how a good local social network would work, but getting it started is the hard part. I’m in a pretty unique position, in that I could create a pretty good city-based local network for Wellington, Dunedin, Auckland and Christchurch, using <a href="http://www.zoomin.co.nz/">Zoomin</a>, but I don’t know what I would need to add to get the community that lives at <a href="http://www.reddit.com/r/wellington">/r/wellington</a>, to move over to zoomin and add their uniqueness to my site.</p>
<h1 id="zoomin-as-a-model">Zoomin as a model</h1>
<p>If I could work out how to make a great hyperlocal site, I could hack it into ZoomIn, advertise to the all the users that hit it each day (about 10,000 a day across New Zealand), and show how a hyperlocal site would work. But there are so many apps doing so many different takes on the hyperlocal scene, so many sites that have tried and failed at building local communities.</p>
<p>One thing that I keep coming back to is to make a sort of ‘search engine’ for local data, where you suck in data feeds from Facebook, Twitter, all over the web and put it on one map so that people can see what’s going on nearby. I really thought that would work, but <a href="http://www.spindle.com/">Spindle</a> did a really good job of that, and they never got traction.</p>
<p>You might think that just letting people add their thoughts to the map would create a community, but <a href="http://www.findery.com/">Findery</a> is doing that, and they have the best community-development team in the world, and they’re still not taking off. Maybe you should create a local-Q and A site, but Localuncle and Localmind tried that, creating really nice iphone apps, and there is no community there.</p>
<p>The <a href="http://daviswiki.org/">davis wiki</a> is a great example of what a community-based local site can be, with masses of content for a small college town in the US. So the localwiki project was created, and they’ve created amazing wiki software with a wysiwyg editor, and the ability to tag pages with location information so you can see maps of all the pages in your city, and it has shown pretty good growth, but when you look at some of the bigger wikis, like the <a href="http://sfwiki.org/">sfwiki</a>, they’re still not the heaving communities that the Davis Wiki was. I’m not sure what is missing, maybe the Davis Wiki had a unique opportunity because it was created before there were a lot of other places to forment local community on the web.</p>
<p>I guess that leaves yelp, google maps and foursquare, which are pumping, but are they the Facebooks for cities? I think the closest to what I envisage would be the yelp forum pages. Mmmm, anyway, time to think on.</p>
Ruby-isms2013-08-15T00:00:00+00:00http://bnolan.github.com/2013/08/15/ruby-isms<p>Here are some ruby things that are really gross and you shouldn’t do:</p>
<h1 id="unless-not-nil">Unless not nil</h1>
<p>I was guilty of this back in the day at <a href="http://projectx.co.nz/">projectx</a>, I used to be editing code and instead of understanding what was going on and fixing it, I’d just insert a not at the start, or an unless, or add .nil? at the end.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>unless not thing.nil?
</code></pre>
</div>
<h1 id="mixed-hashes">Mixed hashes</h1>
<p>I hate the new ruby json-style hash literals. They’re like mega gross. And lots of new rails projects on github use them. I’m sorry I’m so oldschool, and I’m sure I’ll eventually come around from my curmudgeonly ways, but if you ever write code like this, you’re doing it wrong.</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="err">key</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="err">value,</span><span class="w"> </span><span class="err">'other-key'</span><span class="w"> </span><span class="err">=></span><span class="w"> </span><span class="err">value</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<h1 id="that-you-can-sendprivate_method">That you can send(:private_method)</h1>
<p>Why is it even private if you can just call it with .send.</p>
The Reddit App Predictor2013-08-14T00:00:00+00:00http://bnolan.github.com/2013/08/14/reddit-app-predictor<p>This is a bit of a stretch, but it occurred to me while using vine, that you could look at some of the culture of <a href="http://reddit.com/">reddit</a> and use it predict what might be a successful app.</p>
<h1 id="animated-gifs">Animated Gifs</h1>
<p>Animated gifs are commonly used as replies in reddit comment threads. It’s a bit of a stretch comparing <a href="https://vine.co/">Vine</a> videos to animated gifs, but they definitely could have been inspired by. Vine videos are about 6.5 seconds long, optionally have audio and you create them on your mobile device (Android, iPhone). Something I don’t think you could have predicted from the rise of animated gifs would be all the creativity of people recording themselves on their phone. Most of the animated gifs on reddit are extracted from youtube videos.</p>
<h1 id="images">Images</h1>
<p>Reddit doesn’t host images, it only allows text posts, so to fill the niche of a reliable image hosting service, <a href="http://imgur.com/">Imgur</a> was created in 2009. It’s a bit of a leap to go from Imgur to Instagram, but it’s the same thing right, sharing images in a social way. I guess something that vine and instagram have in common is the basic mechanic of creating content on the go and then building a social network around that content, with likes, faves, and reposting content.</p>
<h1 id="memes">Memes</h1>
<p>Advice animals, those images that you see posted with text above and text below….</p>
<p><img src="http://i1.kym-cdn.com/photos/images/newsfeed/000/003/062/Courage-Wolf-Pain-is-temporary-Failure-is-forever.jpg" /></p>
<p>Came out of /b/ on 4chan, but found a permanent home on Reddit. A few meme generators have been created, making it easy for people to create and host these memes. I don’t really see memes as having a great chance of becoming the next big mobile social network, but you never know right?</p>
<h1 id="localreddits">Localreddits</h1>
<p>I found this <a href="http://www.reddit.com/r/ModsOfTheRealms">interesting subreddit</a> the other day, it’s ModsOfTheRealms, and it’s a reddit for moderators of local reddits. Local reddits are things like /r/sydney or /r/wellington, subreddits that are designed for citizens of a local town. I really think this localreddit phenomenon could be a good indicator of a need for people to connect locally. Imagine a mobile app designed for people in a city, with the content of the <a href="http://reddit.com/r/wellington">/r/wellington</a> reddit - but with the traffic of /r/pics. I’m not sure what the largest city-based reddit is - <a href="http://reddit.com/r/nyc">/r/nyc</a> has over 47,000 users, so I guess you could study that and see what it’d be like if you made a Wellington-based forum that was as popular as a larger metro one.</p>
Beervana Recap2013-08-12T00:00:00+00:00http://bnolan.github.com/2013/08/12/beervana-recap<p>So, last Thursday at about 3pm, <a href="https://github.com/ezza">Erin Francis</a> had the idea to build a mobile app that we could use to keep track of what beers we had drunk at <a href="http://beervana.co.nz/">Beervana</a>, and to get some rankings on the beers that people drunk, so that you could see what were the most popular beers.</p>
<p>We hacked it up as a rails app in about 4 hours, and it was ready to go the next day. It is available at <a href="http://beervana.herokuapp.com/">beervana.herokuapp.com</a>, and you can see the <a href="http://beervana.herokuapp.com/leaderboard">leaderboard here</a>. The source code to the app is available on <a href="https://github.com/bnolan/beervana">github</a>.</p>
<p>Some lessons from the app:</p>
<h1 id="its-cool-working-on-something-people-use">It’s cool working on something people use</h1>
<p>We put the app together in a few hours, deployed it to heroku, tweeted it, and were retweeted a dozen times. We also emailed it out to everyone at our office, so that they could use it too. Through that we got 165 signups, and about 50 people used the app to record their beervana experience over the two days. It was super cool, everytime you got into the app, to see more people signed up and using it, and see what beers they recommended.</p>
<h1 id="working-on-fluids-means-you-can-do-maths">Working on fluids means you can do maths</h1>
<p>It was cool writing software that worked with a product that you can easily to maths on. For example we could tell you how many beers you’d had, how much alcohol (standard drinks), what volume, how many different breweries. We didn’t get time, but we could also have rendered a histogram of consumption over time to see when Beervana “peaked”.</p>
<h1 id="having-collaborators-is-awesome">Having collaborators is awesome</h1>
<p>Erin and I could both commit to the repo, so we could riff on ideas super fast and try out new things. Because it was just the two of us working on it, all the decisions could be made super fast, and we could just keep going on the app. We did find after about 3 hours, that we needed to take a break and play some pool and talk about something else, but after that, we just wrote down a checklist of things that needed to be finished before we could call it a night and have the app ready to use the next day. The combination of a checklist and a bit of a break from intense coding worked really well, and we polished the app off quite well.</p>
<h1 id="deploying-from-beervana">Deploying from Beervana</h1>
<p>On the Beervana day (we went to the 11am-4pm session), I took my 11” Macbook with me, in case we needed to fix things on the fly. As it turned out, this worked pretty well. I wouldn’t recommend it for any serious app of course (drinking a beer and using the production console is a bit of a no-no), but for the Beervana app it was fine. I broke the deployment briefly for a few minutes by twiddling with the production console, but that was quickly fixed, and we also added a fix that let you add beers on the fly (since it turned out our ‘authoritive beer list’ was missing a dozen beers).</p>
<h1 id="going-out-after-beervana-was-a-bad-idea">Going out after Beervana was a bad idea</h1>
<p>Beervana was great, and I had an awesome night, but I really should have stuck with my original plan of going straight home from Beervana and having an early night. Staying out late after a day of samping craft beers did not end well for me, and my Saturday was a pretty lazy day of lying in bed and watching movies.</p>
<h1 id="future-of-the-app">Future of the app</h1>
<p>I’m going to talk to Dom at Hashigo and see if he’d be interested in me setting up a similair app for the Pacific Beer Expo. It’d be cool if some more beer festivals used the app (which is basically an event-specific Untappd), it’d be great to see the app being used at the Great Australia Beer Spectapular for example.</p>
<h1 id="build-something-people-want">Build something people want</h1>
<p>There were a few big eye-openers for me in building the Beervana app. One, you can build really fast when you now exactly what you want it to do. Two, It’s awesome building something that people use. I build way too many apps that no-one wants, and thus, no-one ever uses. Three, working with collaborators is awesome. Four, Heroku proves yet again that it’s the best way to deploy quick little apps.</p>
Geoblogging2013-08-07T00:00:00+00:00http://bnolan.github.com/2013/08/07/geoblogging<p>I think geoblogging might become a popular thing. It’s pretty nerdy, and whenever I research the larger social networks, no one really seems to care very much about location when they create content, but I think it might become a thing. I think the most popular geosocial network at the moment is foursquare, and there is a lot of microcontent being created there, hints and tips on places.</p>
<p>People also create foursquare lists, which are awesome maps of places (venues in the foursquare parlance) to visit across cities and countries. I think foursquare is the geosocial network that I wanted to build back in 2006 when I started building zoomin. It’s like flickr but with places.</p>
<h2 id="findery">Findery</h2>
<p><a href="http://findery.com/">Findery</a> is <a href="http://twitter.com/caterina/">Caterina Fakes</a> attempt to build ‘flickr around places’. It’s been going for over a year, and has a bit of funding and about 15 employees. It’s a cool idea, but whenever I use it, I find that it’s not very addictive to use, and I wonder if they just haven’t cracked what it is that makes a site about places really sticky and fun to use. You can write a post about anything on the map, and it uses google places api so that you can search for most businesses and places around the world. I think Findery might get interesting once it has an API available.</p>
<h2 id="hi">Hi</h2>
<p><a href="http://sayhi.co/">Hi</a> is <a href="http://twitter.com/craigmod">Craig Mods</a> “Narrative mapping” site. It’s been built by his team in Tokyo, it’s got the idea of sketches - where you make a few notes about a place, and then people can ask you to expand upon an idea and write more. One thing I find a bit weird about hi, is that it seems you can only write about your current location (it asks for permission to access your GPS), it doesn’t seem that you can write about different places. It also doesn’t have groups or maps, so you can’t make a few posts and mash them together into a little map.</p>
<h2 id="maptia">Maptia</h2>
<p><a href="http://maptia.com/">Maptia</a> isn’t launched yet, but they have an awesome backstory. They’re an english team, that started out in the UK, attending start-up chile, got into techstars in seattle, then moved to a surf-beach town in morocco to live as cheaply as possible and keep cranking on their idea. They have a great twitter presence, lots of people are interested in what they’re doing. But they’re still in private beta, so I haven’t had a chance to see what they’re building.</p>
<h2 id="subtract">Subtract</h2>
<p>Subtract is what I’m working on. It’s a tool for creating mini-blogs about places on your phone. I created it after our trip to the rainforest north of Cairns, where I didn’t have a laptop with me, but I wanted to created a record of our trip, so that I could share the experience with my friends, and have something to point at if anyone I know is visiting that part of the world in the future. It’s a mobile web app, written using rails and deployed on heroku, and it’s my current 1-day-a-week project. I’m modelling it on Tumblr, which has a great mobile content creation experience. The way I’m using it at the moment, each blog you create only has 1-10 entries, for example I’ve created a blog called “Places in Petone”, another one “Newtown, it’s a bit shit” and “Our Cairns trip”. Then you created ‘posts’, and each post is associated with a place. Because you have all this geodata, the app can show you blogs that are near where you are, and recent posts that are happening near you. You can also see a blog as a chronological list of posts, or you can see all the posts on a map.</p>
<p>I did the first cut of subtract in backbone.js, but didn’t find it very easy to riff on and try different ideas, so I’ve re-written it as a pure rails app. I think I’ve got the content creation nailed, it’s pretty easy to create a cool little blog, but the display of the posts is the hard part. I like it when you have a simple content creation tool, and it creates something really nice looking that you’d be proud of showing your friends, you put in a little work and get a nice reward, and the way I display posts at the moment isn’t doing that for me. But I’ll keep riffing on it for a few weeks and see where it ends up.</p>
Turbolinks instead of backbone2013-08-06T00:00:00+00:00http://bnolan.github.com/2013/08/06/turbolinks-instead-of-backbone<p>I read this <a href="http://www.mattdeleon.net/">interesting article</a> by Matt De Leon, about using turbolinks and server-side generated javascript with Rails to build a nice fast app. He is a backbone developer, and found that he could build a nicer app using turbolinks than he could using Backbone.</p>
<p>I’ve found a similair problem. I’ve been working on my “microgeoblogging” tool <a href="http://subtract.in/">subtract</a>, and because I want it to be a nice responsive mobile app, I did the first cut using backbone.js and a minimal rails app that ran the backend. This came together pretty quickly, I knocked it all up in coffeescript and using my <code class="highlighter-rouge">capt</code> build tool.</p>
<p>But the problem was I got to a point pretty quickly where I needed to rapidly change the design, layout and user interface of the app. I built the tool I was imagining, and it was boring to use, it didn’t create the pretty little blog that I was hoping to create, a blog that showed our trip to Cairns and Sydney last month.</p>
<p><img src="/images/subtract-1.png" />
<cite>The most recent iteration of the mobile blog theme</cite></p>
<p>And iterating on a backbone.js app is a whole lot slower than iterating on a rails html view. So I threw away the backbone app for now, and reimplemented everything in rails 4. I had to add a polyfill to get turbolinks running on chrome for android, and I’m not sure that it’s working correctly on ios mobile yet, but even just using proper caching and forms that submit using ajax, the mobile app is pretty fun to use, and it’s been a lot easier to riff on the code and come up with new ideas.</p>
<p>I’m currently at the stage where I just need to keep looking up ideas on <a href="http://dribbble.com/">dribbble</a>, trying them out on the two little blogs I’ve created so far (one for places I like in Petone, and one recording our trip to Cairns), and once I start to make something that is cool and I want to show to other people, then put the app out there.</p>
Branch auto deployment2013-07-25T00:00:00+00:00http://bnolan.github.com/2013/07/25/branches<p>Here’s a good idea. If you worked at a medium sized consumer start-up (say Flickr before yahoo bought them), it’d be awesome that if every time you created a git branch - your CI system automatically created a subdomain for that branch and deployed your code there.</p>
<p>So for example - you run:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git branch new-map-style
git commit -a -m "all the commits"
git push
</code></pre>
</div>
<p>Assuming your specs pass in CI, your CI server would create:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>http://new-map-style.staging.yourapp.com/
</code></pre>
</div>
<p>And deploy your branch there. So that you could post a link to your company message room and say “hey check out my new feature”.</p>
The Official Ben Nolan2013-06-27T00:00:00+00:00http://bnolan.github.com/2013/06/27/official<p>I must point out to my readers, that I am not the Official Ben Nolan.</p>
<p><a href="https://soundcloud.com/ben-nolan-official/">This guy is</a>. Check out his soundcloud page if you like house music.</p>
A more social version of Google Maps2013-06-19T00:00:00+00:00http://bnolan.github.com/zoomin/2013/06/19/zoomin-background<p>For those that don’t know, I’m the owner of <a href="http://www.zoomin.co.nz/">ZoomIn.co.nz</a>. And I thought I’d share some background:</p>
<p>I started zoomin back in 2005, before Google Maps released in New Zealand. Back then there were no mapping apis, so we rolled our own javascript api (using a tool called ka-maps) and created our own map tiles (using data from a nz provider and rendered using mapserver). I originally had the idea of creating a site that was like <em>‘flickr for maps’</em>, where you could find businesses, tag them, categorize them, and add them to groups where people could discuss could places and recommend things to each other.</p>
<p>I started building this as a site called guide.org.nz, and got some good traction with just the maps, even without any businesses or other places. I then met a guy called John Clegg who invested in the company and we renamed the maps to zoomin.co.nz. We got a good lead on wises and google maps in New Zealand, so we started getting a bit of traffic. We then came up with a cool URL addressing scheme, so that a map of leeds street, te aro, would have the map url /wellington/te+aro/leeds+street/. Google then came along and indexed this url scheme and we got some good google juice.</p>
<p>We then added support for tags, comments, photos and groups. So people can create a new business on the site, add photos, use tags to categorise it, and then add comments to record their experience. This was in 2006, and we were getting good traction with the site, and the bulk of the comments and groups on the site were created in 2006/2007.</p>
<p>I then left zoomin at the end of 2006 to go live in Germany, and John and the team kept zoomin running. Sadly, after 2007, the amount of social activity on the site went into a bit of a backwards slide. I’m not sure if this was because different revisions of the site made it too hard to discover the social content, or whether competing sites (menumania, google maps, panoramio) did such a good job of the social aspect of zoomin that people didn’t see the need any more.</p>
<p>Anway - fastforward to 2012 and I got in contact with John and I convinced him to sell me the site. So since mid-2012, I’ve been slowly chipping away at bringing the design of the site up to date, and trying to pump some life back into the site.</p>
<p>So, this is my question, do you think there’s still an opportunity to build a social network around maps and businesses, or is this niche too well served by foursquare, yelp, localist and facebook places?</p>
<p>I’d really love to get the groups pumping again, get people creating little communities around mountain bike tracks, or craft beer bars, or family friendly restaurants. That’s my ultimate goal for zoomin, to be a more social version of google maps.</p>
Aggregator for local restaurants and bars2013-05-07T00:00:00+00:00http://bnolan.github.com/2013/05/07/venuefeed<p>I stumbled across this idea the other day, when I was wondering what was going on in town, so as you do, I went to the facebook page for hashigo zake, then bebemos, then little beer quarter, then goldings free dive. And I thought, there must be a better way to do this.</p>
<p>So, short of adding every single bar and restaurant in town as a facebook friend, and then having my newsfeed spammed with hundreds of updates, I decided to build a tool that shows updates from all the bars and restaurants in Wellington.</p>
<h1 id="httpwwwvenuespw"><a href="http://www.venues.pw/">http://www.venues.pw/</a></h1>
<table>
<tbody>
<tr>
<td>I’m really interested in this idea. The vision I’m working towards would have a filter for bars</td>
<td>cafes</td>
<td>restaurants, the ability to sort by distance or time (so you can see what’s been posted nearby you), and a search function so you can look up if anyone is selling <em>chilli</em> or <em>stout</em>.</td>
</tr>
</tbody>
</table>
<p>I think this could be a really great tool, and if I can get it working well in Wellington, I’ll try get it going in the Mission SF and Melbourne next.</p>
Competitive landscape in trip planning sites2013-04-04T00:00:00+00:00http://bnolan.github.com/placepinner/2013/04/04/competitors<p>I’ve been working on <a href="https://www.placepinner.com/">placepinner</a> exclusively lately, to the expense of my other web property, <a href="http://www.zoomin.co.nz/">ZoomIn</a>. ZoomIn is still making a little bit of adsense revenue though, so it’s ok just to tick over in the background.</p>
<p>So since I’m investing so much time in my trip planning website (since that’s what placepinner will eventually be), I thought I should do a bunch of research on other trip planning sites and learn what they do well, what they lack at and try and work out exactly where the niche is that I want to occupy with Placepinner. I’d love to build the be-all singing-and-dancing trip planning tool from nam that did everything to everyone, but I’m one guy working 20 hours (one full day plus a few evenings) a week, so I have to be very particular with what features I’m going to try and implement. So, here’s a run down of a few sites that do trip planning and what I think of them.</p>
<h2 id="tripit">TripIt</h2>
<p>Great site, doesn’t deal with bars, museums, restaurants, ‘things-i-might-do’. It’s only for confirmed things that have dates or reservations. Great tool, people love it.</p>
<h2 id="kayak-trip-planning">Kayak trip planning</h2>
<p>Very similar to TripIt, doesn’t have the ability to add places i might-want-to-go-to.</p>
<h2 id="trippy"><a href="http://www.trippy.com/">Trippy</a></h2>
<p>Great site. Has gone for the pinterest-of-travel angle. It has a bookmarklet for grabbing photos off website and create a travel wall of places you might want to go. Doesn’t seem to be widely used to bookmark what I’m focussing on (eg bars, galleries, cafes, public spaces). Doesn’t do much with their geodata, eg no ‘what’s nearby links’, no iphone app for exploring the places you bookmarked when you’re actually in a city. This seems like my closest competitor, since they do actually gather geodata, and it would be easy for them to pivot to creating personal city guides based on their data.</p>
<h2 id="dcovery"><a href="http://www.dcovery.com/">Dcovery</a></h2>
<p>This is a great iphone app that has a really good bookmarklet. The bookmarklet is actually better than placepinner in that it looks really nice, and handles multiple places in one webpage (eg reading an article listing places to go in bangkok, rather than the foursquare venue page for a restaurant in bangkok) - but it’s worse than placepinner, in that it doesn’t automagically grab the map location or address on pages that have the address. They also don’t have any web front-end, or ability to edit places and add notes about places. Editing places and adding notes is something that’s coming in the next version of their iphone app, and they have become very popular on the iphone app store. They also have a business model, in that the app is a paid-app (99cents on special at the moment). Great competitor, they’re not in a hurry to go android, so maybe I’ll just duplicate them for android. ;)</p>
<h2 id="rego"><a href="http://www.regoapp.com/">Rego</a></h2>
<p>Just linking these guys because they got a bit of <a href="http://www.macgasm.net/2013/03/27/mans-map-app-blows-up-after-he-accidentally-names-it-butt-crack/">press recently</a>. Haven’t tried out their app since my ipad doesn’t run ios 6, but it looks like a nice tool. You can’t add places by name (searching via google places or foursquare), so it’s not hugely useful as a trip planning tool, but it’s got a nice looking interface, and they understand the potential as a trip planning tool so you can’t discount them.</p>
<h2 id="wanderfly"><a href="http://www.wanderfly.com/">Wanderfly</a></h2>
<p>Wanderfly is an ajax-heavy web app for planning your trips. However, it falls in the trap that a lot of trip planning sites do, in that it only recommends places that already exist in it’s database. It has no easy way to add new places to your trip. It does a good job of recommending places, but a lot of sites do a good job of recommending places, and I still think there is a need for an app that easily lets you grab all these recommendations and jam them together onto one map. Wanderfly doesn’t do this yet, but maybe it will.</p>
<h2 id="foursquare"><a href="https://foursquare.com/">Foursquare</a></h2>
<p>The almighty foursquare. With an awesome venue database, a bookmarklet for saving places (that’s not quite as good as the placepinner one, but is pretty good), and good mobile apps. Foursquare could totally own trip planning, but it’s not their sole dedicated focus at the moment, so they don’t solve it that well. eg, viewing places on my todolist on mobile isn’t that easy to do. To be honest, you could most of what I’m imaging placepinner to be as foursquare add-ons, eg a list of cities in which you have places to-do and not visited,</p>
What I like about Android dev2013-03-20T00:00:00+00:00http://bnolan.github.com/placepinner/2013/03/20/what-i-like<p>I’ve been doing about two weeks of android development now, on and off. It’s been interesting. The things I like:</p>
<h2 id="eclipses-autocompletion-and-code-highlighting">Eclipses autocompletion and code highlighting</h2>
<p>The code intellisense that eclipse provides is really great. I like how it compiles on the fly and shows unused and uninitialized variables, and the autocompleter works really well (coming from a ruby world, it’s a pretty nice feature of a strongly typed language).</p>
<h2 id="sensible-exceptions">Sensible exceptions</h2>
<p>When you try and do some network activity on the main thread, android throws an exception. You can disable this, but you really shouldn’t, since all it’s doing is forcing you to make a responsive app.</p>
<h2 id="asynctask">AsyncTask</h2>
<p>This is a really nice little class that starts up a new thread, runs some code in the thread, then rejoins your main thread and runs some other code in the main thread. That means that you can easily do some network activity (or anything else slow), and easily put the result back into your main UI code. Obviously you shouldn’t try and manipulate your UI from two threads at once, so the <code class="highlighter-rouge">onPostExecute</code> code that runs in the main thread is cool.</p>
<h2 id="nice-animations-and-performance">Nice animations and performance</h2>
<p>Coming from a phonegap background, it’s nice all the little built-in animations that come with android (for example moving between activities), and the performance of list views and page rendering is much quicker than anything you could hope to do with phonegap.</p>
<p>Things I don’t like…</p>
<h2 id="eclipse">Eclipse</h2>
<p>It’s still an ugly troll. And it’s one monolithic window, you can’t break it up and spread it around multiple monitors.</p>
<h2 id="the-android-emulator-is-slow">The Android emulator is slow</h2>
<p>I’ve been told to use the intel-based (atom) emulator, using the Intel VT-X extensions, I haven’t got around to trying that yet (I do most of my testing live on my phone), but the default ARM-based emulator is slow as a dog.</p>
<h2 id="having-to-support-23">Having to support 2.3</h2>
<p>If you look at the <a href="http://developer.android.com/about/dashboards/index.html">android versions dashboard</a>, you’ll see that you have to support Android 2.3 (API version 9) to get more than 50% compatibility with Android devices. 2.3 is okay, and most of the good features you want are backported by Google so that you can use them. I’m still learning what is and isn’t available, but at the moment I’m just using list views, text views and layout components and don’t need to do anything fancy.</p>
<h2 id="theming">Theming</h2>
<p>I’m not sure how theming works at the moment, for example <a href="http://www.twitter.com/ristari">@ristaris</a> 2.3 device has a very different theme to my 4.0 holo themed device. I’m not sure how I’m meant to make UI that looks native on both devices, or whether to make my own style toolbars that don’t really look the part on Android, I guess I’ll find out.</p>
<h2 id="potential-revenue">Potential revenue</h2>
<p>From reading different Android forums (and my own experience with the Rankers app), it seems like there’s very little revenue to made from advertising networks, or even selling pro versions of apps on Android. The iOS store is a lot more lucrative. Sadly, there’s already a really good app that does trip planning / place bookmarking for iOS (it’s called <a href="http://dcovery.com/">dcovery</a>), and I really want to make an Android app, since I’m an Android fan myself. I’m just not sure how I’m going to make a return on all the work that I’ve put into placepinner so far. Maybe sell the app, maybe only allow 100 bookmarks for free users, then encourage people to upgrade to pro. Something anyway.</p>
<p>It’s been an interesting time, I’ve also done a bunch of work on the web-site part of Plcaepinner (it’s still not ready for people to see, but it’s getting there). The three things I need to finish before I can promote placepinner are 1. Chrome extension (for bookmarking places), 2. Website (marketing site basically), 3. Android app. Then I can start to show people what I’ve been working on for the last few months.</p>
Working on an Android app2013-03-13T00:00:00+00:00http://bnolan.github.com/placepinner/2013/03/13/android-app<p>I’ve started working on a native android app for <a href="http://placepinner.com/">placepinner</a>. I decided to go this track after trying out a quick phonegap app using ratchet and backbone.js. Even though it’s super easy to develop, and you can do some quite nice UI with CSS, I couldn’t help but notice how laggy scrolling was using Phonegap.</p>
<p>So I’ve decided to go native. I’m also taking this as a good opportunity to try out Java (which I’ve never used in anger before), and learn some android mobile development (in case the ruby on rails job scene dries up).</p>
<p>My experience so far:</p>
<ul>
<li>Eclipse is an ugly tool</li>
<li>Java is a sweet language</li>
<li>The android emulator is much slower than the ios simulator</li>
</ul>
<p>It’s a bit funny doing all the typecasting and type coercion that’s required to pass stuff around. For example in ruby:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>JSON.parse(IO.readlines('http://placepinner.com/places.json'))
</code></pre>
</div>
<p>Versus this java code to do part of the same…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>InputStream stream;
try {
URL url = new URL(urls[0]);
URLConnection urlConnection = url.openConnection();
urlConnection.setConnectTimeout(5000);
stream = urlConnection.getInputStream();
} catch (Exception ex) {
Log.i ("Exception!", ex.toString() );
return null;
}
String str = "";
try{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int length = 0;
while ((length = stream.read(buffer)) != -1) {
baos.write(buffer, 0, length);
}
str = new String(baos.toByteArray());
} catch (Exception ex) {
Log.i ("Exception!", ex.toString() );
}
return str;
</code></pre>
</div>
<p>We’ll see how far I get. I’m using <a href="http://www.dcovery.com/">dcovery</a> as inspiration for this app. They have a great travel planning bookmarklet and app for IOS. I’m trying to do the same thing for Android, but I’ll probably also have a website (Dcovery doesn’t have any website trip planning tools, so you can’t review / edit your trip on your PC), so that you can view and edit places you have bookmarked, and eventually maybe split places you have ‘pinned’ into seperate days (actually plan a trip day by day).</p>
<p>But yeah - android for now.</p>
Mobile Released2013-03-06T00:00:00+00:00http://bnolan.github.com/zoomin/2013/03/06/mobile-released<p>I released the first iteration of a mobile site for ZoomIn last night. Try it out on your iphone or android phone and you should get a mobile-designed stylesheet. It’s not complete, the recent / popular and group pages in particular aren’t done very well, and I want to add a view that shows you ‘what is nearby’ (using the html5 geolocation api), but it’s a first cut, and the place pages in particular are a lot more usable on mobile than they were before. I haven’t tested it thoroughly on iphone yet (I don’t have the simulator installed on my home laptop), but I’ll probably be doing more mobile work over the coming week, so things will slowly improve.</p>
<p><img src="/images/zoomin-mobile.png" />
<cite>A preview of a places page on the iphone simulator</cite></p>
Responsive Design for Mobile2013-03-01T00:00:00+00:00http://bnolan.github.com/zoomin/2013/03/01/responsive-design<p>I’ve been working on the ZoomIn mobile view, on and off for a few months now. I initially started by creating a :mobile format in rails, so that I could render a totally different view for mobile users. Eg <code class="highlighter-rouge">layout.mobile.erb</code> and <code class="highlighter-rouge">show.mobile.erb</code>. This worked pretty well, and I got off to a flying start, styling up the mobile view and copying view code across from the desktop view.</p>
<p>This worked well for prototyping, but as I spent time redesigning the rest of the site, I realised that there are a <em>lot</em> of views in ZoomIn, and duplicating each one in a mobile view would take forever, and double my maintenance costs. So instead, I decided to go for responsive design, where I serve a different stylesheet for mobile devices. I’m not using media queries at the moment, I’m instead detecting mobile devices based on the user agent, but I think I’ll change this to use screen-width media queries and some javascript to hide / show bits of the UI that aren’t working in mobile yet.</p>
<p>The biggest problem at the moment is that I haven’t built in support for all the popovers that are used for login and most of the interactivity with the site. So it’s a ready only view of the app at the moment. I’ll probably wait and just implement facebook auth for mobile and skip support for the current username / password login. One of the things that has worked well has been using google maps and streetview on mobile, they both run really nicely on my nexus s, which isn’t a super powerful phone. I was going to use leaflet and cloudmade maps since I have experience of that on mobile, but it seems that the google maps implementation is really nice on touch devices.</p>
<p>For design, I’ve been copying lots of bits of UI from <a href="http://maker.github.com/ratchet/">ratchet</a>, so I’ve got a nice blue header bar and some good looking buttons and form elements.</p>
Trip Planner2013-02-27T00:00:00+00:00http://bnolan.github.com/placepinner/2013/02/27/trip-planner<p>I’ve been working on a trip planner, on and off for quite a long time now. Probably since back in 2008, when I started work on ‘weheartplaces’. The original idea was to make a site for bookmarking places you want to go, like delicious, except for places. Basically you’d use it to make a list of places you might want to go one day. <a href="http://www.foursquare.com/">Foursquare</a> does a pretty good job of this with their ‘want to go’ feature. And <a href="http://www.trippy.com/">trippy</a> does a pretty good job of being pinterest for places, you can create a wall of all the pretty places you want to visit one day.</p>
<p>Me personally, I still want to create a chrome extension, that lets you quickly save the current page to your travel map. I’m not sure what the next step should be though, whether to create a fully fledged travel planner where you can divide places into days (I’ll go to volkspark on tuesday and gorlitzerpark on wednesday), or whether just to make a minimal ‘places i’ve starred nearby’ app for iphone and android (using ratcher and backbone.js and phonegap).</p>
<p>So yeah, I haven’t decided where exactly I’m going with this project at the moment, and ZoomIn is taking up most of my free development time (especially now that its growing, slightly, week on week) - but watch this space, there might be a cool chrome extension / travel app coming from me. If I can decide what it is exactly that I want to build.</p>
<p>This post partly inspired by utrip - with their cool travel planning app:</p>
<p><img src="/images/utrip.png" />
<cite>A guide that utrip created for me on a fictional trip to Prague</cite></p>
Groups redesign2013-02-26T00:00:00+00:00http://bnolan.github.com/zoomin/2013/02/26/groups-redesign<p>Tuesday to Friday I work at Powershop, on their large rails app that runs most of the business. They’re a great team and I really enjoy working on such a gnarly rails project.</p>
<p>However - Mondays are my work-from-home-on-ZoomIn day, and this Monday I worked on two things.</p>
<h2 id="rails-23-migration">Rails 2.3 migration</h2>
<p>ZoomIn is written on <a href="http://www.rubyonrails.org/">ruby on rails</a>, starting with version 0.6 back in 2005. Over time it has been upgraded, but when I took over ZoomIn last year, it was stuck back on Rails 2.1. This was a bit of a pain, because some of the extensions I wanted to use (eg Facebook connect) aren’t available for such an old version of rails, and secondly, there are a whole swathe of security issues with Rails apps that were fixed in 2.3.17. So, after updating my gems, copying in the initializers for the new app, I was mostly ready to deploy the new version. I used a git branch to test out the app against Rails 2.3. One of the biggest problems I had was replacing <code class="highlighter-rouge">classic_pagination</code> with <code class="highlighter-rouge">will_paginate</code>, but in doing so, I sped up a few queries and redesigned the pagination links on a few pages, so it worked out to be quite a benefit in the end.</p>
<p>So as of Monday night, ZoomIn is now running on the latest 2.3 version of Rails. Score!</p>
<h2 id="groups-page-redesign"><a href="http://zoomin.co.nz/group/show/8">Groups page redesign</a></h2>
<p><img src="/images/groups-redesign.png" />
<cite>The new layout of the groups page</cite></p>
<p>After getting the rails migration mostly done, I was fiddling around with the <a href="http://www.zoomin.co.nz/group/list/">groups</a> functionality in ZoomIn. This has been a part of the site that I was always very excited about. I had modelled the groups on Flickr groups, which are a very social part of that site. So groups have a name, a description, an owner, and members, places and topics. My first priority was to update the groups so that they used the new <a href="http://twitter.github.com/bootstrap/">bootstrap</a> theme that I had applied to the rest of the site. This was pretty straightforward.</p>
<p>Next up, I was looking at a group that I created (<a href="http://zoomin.co.nz/group/show/8">places I have lived</a>) and decided that it would be cool to add comments to each of the places that were included in a group. So, I made it that under each place that is listed as part of the group, it looks for comments on that place, that were made by a member of the group. If a comment is found, it’s then displayed under the place, so that you can get a bit of a narrative going on of why that place is part of the group.</p>
<p>Next up is to work on Facebook integration I think, since people seem to have problem with the current authentication on ZoomIn. Also it’d be good to make it easier for businesses to add or update (or even promote?) their business on ZoomIn, since I get a lot of emails from people that can’t work out how to add places. But that’s for another day…</p>
ZoomIn Analytics2013-02-21T00:00:00+00:00http://bnolan.github.com/zoomin/2013/02/21/zoomin-analytics<p>The lifeblood of a website is it’s traffic. If no one is using your site, it’s basically dead. One of the things that interested me most when I was considering purchasing ZoomIn, was it’s traffic stats. On a typical Monday, ZoomIn has around 10,000 visitors, 85% of which are from New Zealand.</p>
<p><img src="/images/zoomin-traffic-over-time.png" />
<cite>Traffic of Zoomin from Jan 1st 2007 to today</cite></p>
<p>This doesn’t make ZoomIn a top 100 website in New Zealand in terms of rank, in fact it comes it at number 660 according to the Alexa ranking. That’s even further down the list than <a href="http://www.zenbu.co.nz/">Zenbu</a>, a similair site that has been around for about as long.</p>
<p>However, those visitors every day are interested in New Zealand, and are interested in maps, so they’re a really great sounding board to try new ideas and try and find one that sticks. Something that I noticed early on (back in 2006) in ZoomIn, was that every time we made a release, we would get a slight bump in traffic. I guess this is because we had interested users, and that whenever they noticed a change to the site, they’d dig around and see what had changed.</p>
<p>I sort of foolishly assumed that I’d get the same little traffic bump every time I do a release to ZoomIn now. Sadly, that doesn’t seem to be the case. I’ve done about 50 releases to ZoomIn in the past 6 months, many small changes, and a few large ones (redesigning to use twitter bootstrap for example), and none of these releases have made a significant difference to visitors, page views, or pages per visit.</p>
<p>I’m working my way down my list of things to do to ZoomIn, hopefully I can report back in 6 months and show a slow growth in the traffic stats above.</p>
Dream Hardware2013-02-20T00:00:00+00:00http://bnolan.github.com/zoomin/2013/02/20/dream-hardware<p>I’ve been thinking of upgrading <a href="http://www.zoomin.co.nz/">ZoomIn</a> to some dedicated hardware. I currently use a 4gb ram VM hosted by rimuhosting in Auckland. They’re a pretty good crowd and the performance has been ok, but IO is especially slow, so anything which hits disk is slow. I’ve fixed this by caching expensive things (for example the json used in the new search sidebar) in memcached, and adding indexes wherever <code class="highlighter-rouge">explain analyze</code> showed a sequential scan, but it’s annoying to have to do so much performance work in production, when performance is perfectly acceptable in my development version (running on a SSD and 4gb of RAM). And so I’ve been thinking about moving to some dedicated hardware - maybe something like this:</p>
<ul>
<li><a href="http://www.dell.com/nz/business/p/poweredge-r410/fs">Poweredge R410</a> $1499</li>
<li><a href="http://www.ascent.co.nz/productspecification.aspx?ItemID=401840">16gb RAM</a> $501</li>
<li><a href="http://www.ascent.co.nz/productspecification.aspx?ItemID=404257">120gb SSD</a> $176</li>
</ul>
<p>And then the monthly costs, hosting with <a href="http://www.xtreme.net.nz/services/colo.php">xtreme</a>, who have a Wellington based data center - which is important since the majority of ZoomIns visitors are from New Zealand, and things like the autocompleter require ultra low latency.</p>
<ul>
<li>1U Rackspace $199</li>
<li>Extra 20gb intl. traffic $99</li>
</ul>
<p>Which comes in to at least $300 per month, after the capital costs of buying the server. Another option, but I’m not sure of the quality of their services, is <a href="https://my.hd.net.nz/cart.php?a=confproduct&i=0">HD</a> in Auckland. I’m not going to rush into this, but if I get the advertising revenue up a bit on ZoomIn, it might be worth upgrading to some more chunky hardware.</p>
Xapian to TSearch22013-02-19T00:00:00+00:00http://bnolan.github.com/zoomin/2013/02/19/xapian-to-tsearch2<p>When we first wrote ZoomIn, back in the day, the built in search that came with Postgres (TSearch) was implemented as a bunch of stored procedures, and generally sucked. So we used a high performance C++ reverse index search library called <a href="http://xapian.org/">Xapian</a>. Why did we need full text search in ZoomIn? Well, the plan is that people would tag places with keywords, and then you could generate a searchable column on each place, that contained all the tags, the address and any other information we had about the place, so a search like:</p>
<p><em>beer bar wellington</em></p>
<p>Would surface <a href="http://zoomin.co.nz/map/nz/wellington/te+aro/taranaki+street/25/-hashigo+zake/">Hashigo Zake</a> and <a href="http://zoomin.co.nz/map/nz/wellington/newtown/riddiford+street/88/-bebemos/">Bebemos</a>, just using the magic of full text search.</p>
<p>Note that we didn’t use FTS for our address parser, instead we used a bayesian search tool that I wrote from scratch in a bit of gut-driven intuition, and then Nick rewrote after doing the maths to work out exactly how it should work.</p>
<p>Our Xapian integration was pretty cool, we wrote a module for Postgres that exposed Xapian query results as a temporary postgres table (that we could then join with our Places table). But when I took over ZoomIn last year and moved everything onto new infrastructure, I wanted to reduce the number of moving parts, and one of the easiest ways to do that was to remove Xapian. Luckily, <a href="http://www.postgresql.org/docs/8.3/static/textsearch.html">Full Text Search</a> in postgres has come a long way, and the new (built in) text searching functionality is fast enough on the data set I use. So now I just have an <code class="highlighter-rouge">after_save</code> hook that rebuilds the <code class="highlighter-rouge">tsvector</code> column, which updates the index and makes sure we can always search quickly and freshly on any of the 100,000+ places in the ZoomIn dataset.</p>
ZoomIn Fixed Tree Implementation2013-02-18T00:00:00+00:00http://bnolan.github.com/zoomin/2013/02/18/zoomin-fixed-tree<p>I’ve been working on <a href="http://www.zoomin.co.nz/">ZoomIn</a> a lot in the past few weeks. I need to start writing more newsletters about what I’m doing. But anyway, there’s a technical aspect of ZoomIn that I want to highlight, since it’s really great.</p>
<p>Back in 2006, when <a href="http://www.projectx.co.nz/">John</a>, <a href="http://www.indigorenderer.com/">Nick</a> and I were working on the first version of ZoomIn, we were discussing ways of storing the ZoomIn url hierarchy in the database. So for example, we had nodes like:</p>
<ul>
<li>/wellington/</li>
<li>/wellington/newtown/</li>
<li>/wellington/newtown/coromandel-street/</li>
</ul>
<p>And in the database, we wanted to store that newtown was a child of wellington, and coromandel street a child of newtown. The easiest way to do that would have been just randomly inserting nodes into the database and building a tree structure where each node has a parent_id. The only downside to this, is that I wanted a really efficient way of finding all the nodes, comments, or photos, under a node in the tree.</p>
<p>eg - we wanted to be able to do a single query (without any crazy joins) that would return all comments on any streets, places or street addresses under <code class="highlighter-rouge">/wellington/newtown/</code>.</p>
<p>We went away for a while, and bounced over a bunch of ideas, then eventually stumbled on the inet6 datatype that is native to postgresql. So after a bit of research, we worked out that we could use an inet6 as a primary key and foreign key, rails worked well with it, and indexes operated fine as well. So what we ended up doing was something like this:</p>
<ul>
<li>/wellington/ => 1::</li>
<li>/wellington/newtown/ => 1::1::</li>
<li>/wellington/newtown/coromandel-street/ => 1::1::1::</li>
<li>/wellington/newtown/hanson-street/ => 1::1::2::</li>
<li>/wellington/newtown/hanson-street/43/ => 1::1::2::1::</li>
<li>/wellington/berhampore/ => 1::2::</li>
</ul>
<p>And we wrote two helper methods on each Place (a node in the tree), <code class="highlighter-rouge">left</code> and <code class="highlighter-rouge">right</code>.</p>
<ul>
<li>wellington.left => 1::1::</li>
<li>wellington.right => 1::FFFF::</li>
</ul>
<p>So - if for example, you want to find all the comments under Newtown (to show the new <a href="http://www.zoomin.co.nz/nz/wellington/newtown/forum/">forum page</a>), you do a query like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>['select * from comments where place_id between ? and ?', newtown.left, newtown.right]
</code></pre>
</div>
<p>In the end, we actually used a variable sized structure (so we don’t use a whole subnet per node, but instead a specific bit-depth), but the basic idea is as above.</p>
<p>I’m not sure what kind of data structure this is (I think maybe it’s a fixed tree?), but it’s worked excellently for over 7 years on ZoomIn. Combined with the GIST 2-dimensional index that PostGIS supplies, we’ve been able to do most all the queries you want on a geo-social site without relying on denormalizing the data or super expensive mega-joins or subselects.</p>
<p><em>Edit</em> <a href="https://twitter.com/stojg">Stig</a> from SilverStripe informed me that this technique is called a <a href="http://en.wikipedia.org/wiki/Nested_set_model">Nested Set</a>.</p>
Trumpeter2012-06-28T00:00:00+00:00http://bnolan.github.com/2012/06/28/trumpeter<p>I’m not sure if I’ve already shared my rendition of the Imperial March.</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/ZiXNUzFYvmw" frameborder="0">
</iframe>
Setting cookies with selenium webdriver2012-06-08T00:00:00+00:00http://bnolan.github.com/2012/06/08/selenium-webdriver-cookies<p>This took me more googling and experimentation than it should have. If you want to set cookies using selenium webdriver under rails (for your cucumber specs for example), you want to use <code class="highlighter-rouge">add_cookie</code>. For example, we use this in our web_steps to authenticate the user by setting the auth cookie directly:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>visit '/'
page.driver.browser.manage.add_cookie(:name => "auth_token", :value => auth_token)
</code></pre>
</div>
<p>Here’s hoping the next developer trying to do this finds the answer more easily. :)</p>
Streetview2012-06-07T00:00:00+00:00http://bnolan.github.com/2012/06/07/streetview<p>I’ve been playing around with a little social places site at <a href="http://www.weheartplaces.com/">weheartplaces</a>. One thing I’m really proud of is the streetview-based navigation. When you’re <a href="http://www.weheartplaces.com/places/7687">viewing a place</a>, you can spin around, and there’s a little hyperlink over other places nearby. Hover over that hyperlink and you can see the name and address of the place, and click then link to zoom through to that place page, where you can edit the details or comment on the place.</p>
<p><img src="/images/streetview.png" /></p>
<p>It’s a nice little bit of UI. I’ve been tempted to try and get it going against the foursquare api, then you could browse the world via streetview anywhere in the world - but to be honest, google maps already does a pretty good job of streetview.</p>
Colorscheme2012-06-06T00:00:00+00:00http://bnolan.github.com/2012/06/06/colorscheme<p>Taking from a colorlovers <a href="http://www.colourlovers.com/craft/trends/handmade/7897/Duct_Tape_Feather_Earrings">scheme</a>, I came up with this logo.</p>
<p><img src="/images/colorscheme.png" /></p>
<p>I really like helvetica neue, it looks good in medium and light weights.</p>
Division Systems2012-06-05T00:00:00+00:00http://bnolan.github.com/2012/06/05/division-systems<p>A logo I made when I was a teenager for the business I wanted to start at the time.</p>
<p><img src="/images/division-systems.png" />
<cite>Division systems logo</cite></p>
<p>I just realised it looks pretty similair to the logo for my blog these days. :)</p>
Be careful cycling2012-05-18T00:00:00+00:00http://bnolan.github.com/2012/05/18/be-careful-cycling<p>I bought a nice road bike, a Giant OCR2, for my daily commute from Petone to Wellington. I don’t bike on rainy days though, busses on those days.</p>
<p><img src="/images/ocr2.png" />
<cite>Not my bike, but looks the same</cite></p>
<p>I learnt my lesson about not going too fast along the waterfront yesterday. A lady cyclist popped out of nowhere and I jumped on the anchors but still slammed into her hard enough to bend my shifters. So now I have gammy shifters that won’t change up properly, and I need to remove the hoods and straighten the shifters out so that they face forwards again. To be honest it was a bit of a lucky save, I’m 110kg, so if I’d hit an old lady or someone with a pram, it could’ve been a way worse situation. So lessson learnt, hold fire on full-power until I get onto Thorndon Quay, or even better, the old hutt road where there is less traffic and I can really unleash.</p>
<p>Apologies to the lady that I busted into, I’m glad you weren’t hurt stranger lady.</p>
Capt updates2012-05-14T00:00:00+00:00http://bnolan.github.com/2012/05/14/capt-updates<p>Did some quick <a href="/capt/">capt</a> updates today. I bumped the version of jQuery, Backbone and Underscore. I fixed some bugs that new users would have found with a missing <code class="highlighter-rouge">/app/templates/</code> folder, and I (finally) got around to renaming controllers to routers. My next task with capt is to write some specs for it, and write some more examples to show how to use it.</p>
Holiday2012-01-30T00:00:00+00:00http://bnolan.github.com/2012/01/30/holiday<p>There’s been a pretty large gap in my blogging. This is my attempt to get back in the saddle. I spent a few weeks over christmas in the philippines, meeting my partners family, and going diving at Moalboal and Alona beach.</p>
<p><img src="/images/holiday.jpg" />
<cite>Hammock at Moalboal diving resort</cite></p>
<p>I’ve only got my PADI open water, but spent about 16 hours underwater over the course of two weeks. The diving in the Philippines is probably some of the cheapest in the world. One place we went was about $30 per dive including gear rental, boat hire and everything.</p>
<p>Eating was pretty cheap too. A typical night out with drinks and dinner might cost $10-15 for two people. Beer was 50 cents, and when it’s hot and sunny 50 cent beer tastes about perfect.</p>
<p>I’ve set myself up to be in Wellington this year. I’m working with the team at <a href="http://www.youdo.co.nz/">youdo</a>, and enjoying working on a large scale ruby on rails project.</p>
<p>Happy new year!</p>
Big MySql Imports2011-11-07T00:00:00+00:00http://bnolan.github.com/2011/11/07/big-mysql-imports<p>Today I’ve been fighting a 10+ gigabyte mysql import. I’ve come across and solved a few problems that might be handy to someone in the future.</p>
<h1 id="max_allowed_packet">max_allowed_packet</h1>
<p>In your <code class="highlighter-rouge">/etc/my.cnf</code>, it specifies the maximum packet size. I had the problem where some of the insert statements exceeded 1 megabyte in size, so increasing this limit to 64M solved my first problem.</p>
<h1 id="resuming-from-mid-way-through-an-import">Resuming from mid-way through an import</h1>
<p>Next up, I wanted to <code class="highlighter-rouge">cat</code> the sql import, but skip all the tables up to table <code class="highlighter-rouge">xyz</code>. This I accomplished with a little bit of sed magic:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cat dump.sql | sed -n -e '/`some_table_name`/,$p' | mysql --init-command="SET foreign_key_checks = 0;" database_name
</code></pre>
</div>
<p>Note the <code class="highlighter-rouge">set foreign_key_checks</code> parameter to mysql, this is normally set at the top of your dump file, because we’re truncating the front off the dump file, you need to disable foreign key checks explicitly.</p>
<h1 id="using-pipe-viewer">Using pipe viewer</h1>
<p><a href="http://www.ivarch.com/programs/pv.shtml">PV</a> is a tool to graph your progress. Instead of running <code class="highlighter-rouge">cat</code> and hoping for the best, you get graphical feedback on how long your database restore process will take.</p>
CSV Parser in CoffeeScript2011-10-24T00:00:00+00:00http://bnolan.github.com/2011/10/24/csv-parser<p>I recently needed to be able to parse CSV export from excel, in Javascript. The only way I could find to parse it was a stream-oriented parser, so I came up with this that seems to cope with the majority of strangeness, including multiline cells, quotation marks, and escape and non-escaped cells.</p>
<p>It’s implemented in coffeescript:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class CSV
constructor: (data) ->
@raw = data
@_parse(@raw)
_parse: ->
@rows = []
row = []
cell = ""
offset = 0
mode = CSV.RAW
while offset < @raw.length
ch = @raw.charAt(offset)
adjacent = @raw.charAt(offset+1)
if mode == CSV.RAW
if ch == ","
row.push cell
cell = ""
else if ch == "\r"
@rows.push row
row = []
cell = ""
else if ch == "\""
mode = CSV.ESC
else
cell += ch
else if mode == CSV.ESC
if ch == "\""
if adjacent == "\""
cell += ch
offset += 1
else
mode = CSV.RAW
else
cell += ch
else
throw "Invalid csv parser mode"
offset += 1
@rows
CSV.RAW = 1
CSV.ESC = 2
@CSV = CSV
</code></pre>
</div>
<p>This is a good example of code which will block the browser in a big-time way. For example, if you tried to parse a one megabyte CSV file, you’ll probably crash webkit mobile devices, and get script timeout errors in slower javascript implementations. Whoever, because all the looping logic is in a while, it’d be straightforward to make the parser run inside a setInterval loop, and let the browser remain responsive.</p>
Overscore.js2011-10-07T00:00:00+00:00http://bnolan.github.com/2011/10/07/overscore<p>I’ve extracted the non-blocking <code class="highlighter-rouge">.map</code> and <code class="highlighter-rouge">.select</code> implementation that I use in my mobile apps to keep them responsive while processing large arrays. I called it <a href="http://github.com/bnolan/Overscore">overscore.js</a> and it’s available on github.</p>
Javascript Mobile Apps - Tooling2011-09-25T00:00:00+00:00http://bnolan.github.com/2011/09/25/tooling<p>As another part of my series on Javascript mobile apps, I’m going to discuss tooling.</p>
<p>I’ve listed my toolkit before, but here it is again:</p>
<ul>
<li>jQuery</li>
<li>CoffeeScript</li>
<li>Backbone.js</li>
<li><a href="/capt/">Capt</a></li>
</ul>
<h1 id="jquery">jQuery</h1>
<p>I use jQuery for the same reason everyone else does. It’s well implemented, fast and fun to use. The well implemented part becomes important when you’re dealing with fiddly implementation details of <code class="highlighter-rouge">$.ajax</code> calls. Being able to set headers, read headers and hook the progress events of the ajax object is important to a good mobile app. I use etags when checking for updates in my apps, specifying headers in a convenient hash shorthand is the kind of small detail that jQuery covers so well.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$.ajax {
url : "http://example.com/endpoint/"
dataType : 'json'
timeout : 25000
headers : { 'If-None-Match' : localStorage.getItem('etag') }
}
</code></pre>
</div>
<h1 id="coffeescript">CoffeeScript</h1>
<p>There are pros and cons of Javascript. The biggest downside I see is that if you’re working on projects that aren’t 100% coffeescript, you’re going to end up with ugliness. You either want to be able to convert all your code over to coffeescript (with the exception of 3rd party libraries), or you want to leave it all as javascript. Doing a combination is going to only cause heartbreak.</p>
<p>If you are sold on CoffeeScript though, it’s a fantastic tool. Using the fat arrow <code class="highlighter-rouge">=></code> to ensure class methods are called with the right <code class="highlighter-rouge">this</code> scope is an awesome timesaver. The biggest problem with CoffeeScript is probably debugging. This was easier in older versions of the coffee compiler, which had a 1:1 line to line conversion ratio, so model.js:23 was the same as model.coffee:23. Now, unhappily, your line numbers won’t match up, but as long as you can view the javascript output and find the error, you can easily work back and find out which line of coffeescript blew up on you.</p>
<h1 id="backbonejs">Backbone.js</h1>
<p>If you want a reason why to use Backbone, just view the <a href="http://documentcloud.github.com/backbone/#examples">apps using Backbone</a> list on the backbone site. It’s a fantastic small piece of kit. It adds events, collections and models in a sensible way that will be familiar to any rails developer, but it does it without trying to pervert Javascript.</p>
<p>Basically, if you’re recieving json data from an endpoint and displaying it using Javascript, you should be using backbone or something similar.</p>
<p>I personally, used to make the mistake of trying to be ultra-minimal in my Javascript, avoiding doing heavy computation or having a lot of code over in the client. I’ve changed my way over the past year or two. If you’re struggling to do something without the correct tools, you’re wasting time, and you’ll probably end up with a half-assed solution. Refactor, break your code out into more classes, and increase your lines of code in Javascript.</p>
<h1 id="capt">Capt</h1>
<p>This is my own tool, that I can’t really recommend to anyone else at the moment, since it’s a little bit half-assed and half-finished. But the important thing to do is to have a build tool that you can rely on. You want to be able to use .eco, .less, .coffee and .jst files easily, without having to compile them by hand, or having some makefile you run over and over. Find a build tool that watches your files, recompiles on the fly and includes each file seperately, so that when you get error messages, it’s not just <code class="highlighter-rouge">error in bundled.js:31423423</code>, which is going to be pleasantly impossible to debug.</p>
<p>I’d be keen to hear what other developers use for their build tools.</p>
Javascript Mobile Apps2011-09-24T00:00:00+00:00http://bnolan.github.com/2011/09/24/javascript-mobile-apps<p>I’ve worked on three javascript mobile apps now, one used internally by Lonely Planet, one on the <a href="http://itunes.apple.com/nz/app/rankers-app/id454894632?mt=8">appstore</a>, and another an unreleased site I built myself. I’ve been wanting to write up my notes on development for a while, so since it’s a sunny day and the house is tidy, I’ll put up some notes. First up…</p>
<h1 id="reasoning">Reasoning</h1>
<p>First up, why even use Javascript + XHTML to build your mobile apps? The obvious answer is to get cross-platform support out the box, a well written app will deploy on phonegap to android and iOS with no extra work. Another answer, which is easily dismissed, is the benefits of having a single tool that you can use across more than just your mobile app.</p>
<p>Any business logic you encode in Javascript you can re-use for your desktop client and can be tested on the command line using node.js and <a href="https://github.com/tmpvar/jsdom">jsdom</a>. Where possible, one language is better than two in my book.</p>
<p>Finally, by using javascript, you are free from the tyranny of app-stores. I’ve had apps initially rejected by the appstore, and although they were easily resubmitted and are now selling well, if you’re doing anything “fringe”, that may run afoul of the appstore, having the option to deploy your app over the web as an html5 app is a nice fallback.</p>
Using the new static streetview api2011-09-19T00:00:00+00:00http://bnolan.github.com/2011/09/19/using-postgis-and-static-maps<p>So, Google just released a <a href="http://googlegeodevelopers.blogspot.com/2011/09/quick-and-simple-street-view-with.html">static street view api</a>, which is awesome, since showing a streetview of a place will usually jog someones memory much quicker than a top down view.</p>
<p>The only problem with the API (and it may be fixed in a future revision), is that you need to specify the heading that you want the returned image to be facing. This is a problem, since usually you only have the lat long of the place that you are visualizing, and not the bearing from the nearest road that a google streetview car drove along.</p>
<p>However - in my case, I have a complete set of OpenStreetMap data in my places database, so I could:</p>
<ul>
<li>Look for the nearest street</li>
<li>Calculate the nearest point on the street</li>
<li>Get the azimuth angle between the nearest point and the destination place</li>
<li>Convert radians to degrees</li>
</ul>
<p>And get the correct street view images. See this <a href="http://nz.geonear.com/places/search?q=curry">search for curry</a> on Geonear to see what I mean.</p>
<p>Here’s the code I used (rails code using postgis):</p>
<div class="highlighter-rouge"><pre class="highlight"><code><%= image_tag ".../streetview
?location=#{place.latitude},#{place.longitude}
&size=280x200
&pitch=15
&fov=65
&heading=#{place.street_view_heading}
&sensor=false" %>
</code></pre>
</div>
<p>And in <code class="highlighter-rouge">place.rb</code>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def street_view_heading
street = nearest_street
point = Place.find_by_sql(['select st_closestpoint(?, ?) as geometry', street.geometry, self.geometry])
.first
.geometry
(Street.find_by_sql(['select st_azimuth(?, ?) as azimuth', point, self.geometry])
.first
.azimuth
.to_f * 180 / 3.14159).floor
end
</code></pre>
</div>
<p>And we’re away. Very nice, thanks Google. :)</p>
Lerping LatLngs2011-08-24T00:00:00+00:00http://bnolan.github.com/2011/08/24/lerp-latlng<p>I needed some code for lerping google maps latlongs (for providing a custom streetview navigator). Here’s the function in coffeescript:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>google.maps.LatLng.prototype.lerp = (b, i) ->
lat = (b.lat() - @lat()) * i + @lat()
lng = (b.lng() - @lng()) * i + @lng()
</code></pre>
</div>
Etags, Phonegap and Rails2011-08-23T00:00:00+00:00http://bnolan.github.com/2011/08/23/etags-phonegap-and-rails<p>In the <a href="http://www.rankers.co.nz/">rankers app</a> that I’ve recently finished for a client, I cached the users data offline, but needed a way to quickly and easy to see if the user had the freshed version of the data.</p>
<p><img src="/images/rankers-updates.png" />
<cite>Downloading database updates</cite></p>
<p>To speed up loading time, and enable offline access, I download the list of places (about 350kb) by an ajax call, and then cache the json and the etag of the latest revision in <code class="highlighter-rouge">localStorage</code>. Then when the app launches next time, I send a request for the newest version of the places list, and attach the etag to the request, so that rails can send a not-modified response, and save the users some data.</p>
<p>Here’s the rails pseudocode code I used:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>@places = Proc.new do
Place.find(:all)
end
response.etag = @etag = [
Place.find(:first, :order => 'updated_at desc').updated_at.to_i,
Place.count
]
if request.fresh?(response)
head :not_modified
else
render :action => 'index', :mime_type => 'application/json'
end
</code></pre>
</div>
<p>And in the view:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><% cache(@etag) do %><%= @places.call.to_json %><% end %>
</code></pre>
</div>
<p>This cached the json (since json generation can be pretty slow in ruby), and sends the appropriate header. To handle this on the client side, I used something like this, to cache the places data, and the etag.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$.ajax {
url : "http://example.com/some/endpoint"
dataType : 'json'
headers : { 'If-None-Match' : localStorage.getItem('etag') }
success : (data, textStatus, xhr) =>
if textStatus == "success"
localStorage.setItem('etag', xhr.getResponseHeader('Etag'))
localStorage.setItem('placesData', JSON.stringify(data))
else if textStatus == "notmodified"
data = JSON.parse(localStorage.getItem('placesData'))
else
throw "your toys"
Places.reset(data)
}
</code></pre>
</div>
<p>I was going to implement my own version of etags and <code class="highlighter-rouge">if-none-match</code>, since I thought I might get heisenbugs with different implementations of xmlhttprequest on different mobile webkits, but in the end, I decided not to reinvent the wheel and use what already exists. So far this has worked well.</p>
Detecting phonegap2011-08-22T00:00:00+00:00http://bnolan.github.com/2011/08/22/phonegap-detection<p>With my capt apps, I do most of the development and testing using the desktop safari browser, and then move onto testing on actual devices later on, once I’ve got the app mostly working. I use this fragment of code to detect if I’m running inside phonegap, or if it’s a desktop browser, and then instantiate my app appropriately.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>function onDeviceReady(){
window.app = new Application;
app.start();
}
if(window.PhoneGap){
document.addEventListener("deviceready",onDeviceReady,false);
}else{
$(document).ready(function(){
onDeviceReady();
});
}
</code></pre>
</div>
<p>The window.app style of encapsulating my apps has worked well for me in apps I’ve built using <code class="highlighter-rouge">capt</code> and <code class="highlighter-rouge">backbone</code>, I still haven’t got to the point of encapsulating everything (all classes are copied into the window scope for example).</p>
Duplication detection with postgis2011-08-20T00:00:00+00:00http://bnolan.github.com/2011/08/20/duplication-postgis<p>I’ve written a particularly gnarly query here that I thought I would share. It’s for iterating over a collection of points that have names, and identifying points that likely candidates for merging.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>select
name, floor(length(ST_ExteriorRing(st_box2d(st_extent(geometry)))) * 1000000) as length, count(id)
from
places
group by
name
order by
count desc;
</code></pre>
</div>
<p>Basically, it selects places with similair names, and returns the length of the bounding box surrounding those places. You can use this as a basic test to see if those places should be merged. There a tonne of situations in which this won’t work, but to get you out the gate, it might be helpful.</p>
Google doesn't get the web2011-08-19T00:00:00+00:00http://bnolan.github.com/2011/08/19/google-doesnt-get-the-web<p>It must drive the google search team crazy, when they look at the sort of html the rest of the google organisation spits out. Take for example - this <a href="http://maps.google.com/maps/place?cid=3923961084232206133">google places</a> page. There is no html, no metadata, no microformats, no schema.org support. It’s basically impossible to index. How can they run the worlds biggest web crawler, but generate such obtuse and impossible to index code themselves? It’s astounding I say!</p>
Phonegap2011-08-15T00:00:00+00:00http://bnolan.github.com/2011/08/15/phonegap<p>I’ve got two apps on the go using Phonegap, and this is just a quick post to say that I can highly recommend the mobile web as a development platform. My list of tools / libraries that I use:</p>
<ul>
<li>Capt for building the project and compiling / watching intermediate languages</li>
<li>LESS.js for css</li>
<li>Coffeescript</li>
<li>Backbone.js</li>
<li>Underscore.js</li>
<li>My responsive map/reduce underscore.js helpers (yet to be released)</li>
<li>Leaflet for touch-based maps that are GPU accelerated</li>
<li>JSON.js from douglas crockford</li>
<li>jQuery, although I may move to using zepto later on</li>
<li>Phonegap for deploying to Android and iPhone</li>
<li>Makefile for packaging and bundling</li>
<li>Jasmine for speccing</li>
<li>Latlon.js for distance and bearing functions</li>
<li>iScroll.js for fixed position headers and footers</li>
<li>Cross-origin ajax requests to allow local development while accessing remote servers</li>
<li>ECO templates by Sam Stephenson for in-app rendering</li>
<li>JST templates from underscore.js for static html</li>
</ul>
<p>It’s a pretty nice toolkit. I need to do some more work on capt and put up a sample app so that other people can use my toolkit, but yeah, coffeescript, backbone and less.js are a great way to build mobile apps.</p>
Payroll2011-07-28T00:00:00+00:00http://bnolan.github.com/2011/07/28/payroll<p>When you’re using <a href="http://xero.com/">xero</a> and <a href="http://minutedock.com/">minutedock</a>, you get used to having nice quality tools. It’s like having chisels that were shaped by <a href="http://twitter.com/storminwalker">master</a> <a href="http://twitter.com/nikz/">craftsmen</a>. And then there’s your payroll package…</p>
<h1 id="payroll">Payroll?</h1>
<p>Payroll is the software that takes money from my company account, pays the tax man, deducts my taxes as an employee, then pays me in my personal account.</p>
<h1 id="ipayroll">iPayroll</h1>
<p><img src="/images/payroll-ipayroll.png" /></p>
<p>I’ve always used iPayroll. Giles Crisp is a great contact there (tell him Ben sent you), and provides great service. With iPayroll, you authorize them to move the money around and they get access to your company bank accounts. You will be paid every Tuesday without having to lift a finger. Plus it does all sort of HR stuff as well, so if you have employees, you can know know how many days off they should have.</p>
<p>Anyway - Ipayroll is great, but it’s kinda funky looking, and trying to sign up just recently has been a real PITA. Bugs a plenty with the sign-up process.</p>
<p>So I went to the xero <a href="http://www.xero.com/advisors/solutions/payroll/">payroll providers page</a>.</p>
<h2 id="smart-payroll"><a href="http://www.smartpayroll.co.nz/">Smart Payroll</a></h2>
<p><img src="/images/payroll-smart.png" /></p>
<p>Looks awesome. Until you actually look at the screenshots of the real app, not the pretty marketing page. It looks like it was designed in 1901. It makes iPayroll look a perfectly executed piece of modern art. Pass.</p>
<h2 id="flexitime"><a href="http://flexitime.co.nz/">Flexitime</a></h2>
<p><img src="/images/payroll-flexitime.png" /></p>
<p>Looks awesome. The app is very xero / minutedock-ish, so I rang them up on their 0800 number and asked:</p>
<p>“Can I pay you money to take money from my account and move the money around?”</p>
<p>“Err - no sorry”</p>
<p>The Flexitime guy told me what I’d have to do to use flexitime to satisfy my obligations as an employer. Basically, set up an automatic payment, modify the AP as necessary, update flexitime, and remember to log into IRD once a month and upload a report that I download from the flexitime site. That’s a lot of remembering.</p>
<p>If Flexitime had put a ‘we manage everything’ option on their website with a nice CSS3 button for me to press and a well form that let me set up my organisation, I probably would have chosen them as my payroll provider. Oh well, back to iPayroll.</p>
Map Reduce in Javascript2011-07-27T00:00:00+00:00http://bnolan.github.com/2011/07/27/map-reduce<p>Here’s the non-blocking map reduce class that I was talking about. It uses at most 75% of the CPU, so that the browser remains responsive while your job is processed. You can destroy the job at any stage.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class MapReduceJob
constructor: (inputsOriginal, mapFunc, reduceFunc) ->
utilisation = 75
inputs = inputsOriginal.slice()
# todo - replace with a better clone function
inputs = for input in inputsOriginal
input
outputs = []
@interval = setInterval( =>
tzero = (new Date).getTime()
# Process at least one
if inputs.length > 0
outputs.push mapFunc(inputs.pop())
# Process more
while ((new Date).getTime() - tzero < utilisation) and (inputs.length > 0)
outputs.push mapFunc(inputs.pop())
if inputs.length == 0
clearInterval(@interval)
reduceFunc(outputs)
, 100)
destroy: ->
clearInterval(@interval)
delete @interval
</code></pre>
</div>
<p>You use it like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>new MapReduceJob(
['a', 'b', 'c', 'd'],
function(input){
// do something computationally expensive here, like uppercasing the input
return input.toUpperCase();
},
function(outputs){
console.log(outputs);
}
);
</code></pre>
</div>
<p>This is excellent for things like clustering markers on a map, or doing a backbone filtering on large collections. You need to structure your code to be asyncronous.</p>
<p>I’ve got a few things I might do with this code:</p>
<ol>
<li>Port underscore.js to this continuations style, so that you can use all the existing functions. I’ve already ported the <code class="highlighter-rouge">collect</code> and <code class="highlighter-rouge">inject</code> functions.</li>
<li>Add a progress() method to the class.</li>
<li>Add support for webworkers for desktop browsers.</li>
<li>Port backbone.js to use these continuations, so that for example, the sort function doesn’t block when sorting large arrays.</li>
<li>Assuming the mapFunction is constant time, measure the number of inputs to process per interval, then stop checking the date function.</li>
</ol>
<p>It’s pretty interesting stuff, we’ll see how we go.</p>
CORS, Rails and Weinre2011-07-26T00:00:00+00:00http://bnolan.github.com/2011/07/26/thank-me-later<p>Doing mobile development on your Android but can’t get any debugging action going? Add this code (from the inimitable <a href="http://www.tsheffler.com/blog/?p=428">Tom Shelfer</a>) to the rails controller that provides your json views, so that you can serve your app on a different port than you serve your API:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>before_filter :cors_preflight_check
after_filter :cors_set_access_control_headers
protected
# For all responses in this controller, return the CORS access control headers.
def cors_set_access_control_headers
headers['Access-Control-Allow-Origin'] = '*'
headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
headers['Access-Control-Max-Age'] = "1728000"
end
# If this is a preflight OPTIONS request, then short-circuit the
# request, return only the necessary headers and return an empty
# text/plain.
def cors_preflight_check
if request.method == :options
headers['Access-Control-Allow-Origin'] = '*'
headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-Prototype-Version'
headers['Access-Control-Max-Age'] = '1728000'
render :text => '', :content_type => 'text/plain'
end
end
</code></pre>
</div>
<p>Then install the MacOS version of <a href="http://phonegap.github.com/weinre/">weinre</a> and edit <code class="highlighter-rouge">Content/MacOS/Launcher</code> like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>java \
-XstartOnFirstThread \
-classpath $BASEDIR/weinre-ui.jar:$BASEDIR/weinre.jar:$BASEDIR/swt.jar \
weinre.application.GUIMain --boundHost -all-
</code></pre>
</div>
<p>Then launch weinre (webkit inspector for mobile devices), and add weinre to your web app and reload the browser.</p>
<div class="highlighter-rouge"><pre class="highlight"><code><script src="http://192.168.1.77:8080/target/target-script-min.js#anonymous"></script>
</code></pre>
</div>
<p>Sorry this post isn’t very cohesive - but that there is a fantastic stack for debugging standalone apps to access existing rails apps. I’m using it to debug my phonegap version of the <a href="http://rankers.co.nz/">rankers app</a>.</p>
Something expensive2011-07-26T00:00:00+00:00http://bnolan.github.com/2011/07/26/something-expensive<p>I’m clustering 1200 points on the iphone in javascript in an app I’m writing at the moment. On my iphone 2g it takes 1500ms in a worst case. There are two fixes I’m going to do:</p>
<h1 id="quadtree">Quadtree</h1>
<p>I’ll probably end up implementing a quadtree, or a grid-based acceleration structure, since currently the complexity is currently n squared in a worst case. I’ll let you know how I get on with this.</p>
<h1 id="break-something-expensive">Break something expensive</h1>
<p>This is a generic pattern you can use in javascript (here implemented in coffeescript) to maintain responsiveness in your jobs. You create an interval that runs every 100ms, and it will do at most 50ms of work in each loop, until your data has been processed, then terminates the job.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>inputs = [.., .., .., ..]
outputs = []
interval = null
interval = setInterval( =>
tzero = (new Date).getTime()
while ((new Date).getTime() - tzero < 50) and (inputs.length > 0)
input = inputs.pop()
outputs.push(
doSomethingExpensiveWith(input)
)
if inputs.length == 0
clearInterval(interval)
console.log(outputs)
, 100)
</code></pre>
</div>
<p>I might pull that code out into a generalised form (since I could use it throughout the app) and also add support for chaining inputs and outputs:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>inputs.mapReduce(filterFunction).mapReduce(processFunction).mapReduce(outputFunction);
</code></pre>
</div>
<p>Something like that would be pretty cool, but the above code is the simplified version that works for me.</p>
<p>:)</p>
Ajax download progress2011-07-21T00:00:00+00:00http://bnolan.github.com/2011/07/21/ajax-download-progress<p>I’m working on an iphone app that needs 650kb of json data to get started, so I needed a bit of progress display while the data is being fetched. This works on mobile webkit:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>interval = null
$.ajax {
url : "endpoint.json"
dataType : 'json'
xhr : () =>
xhr = jQuery.ajaxSettings.xhr()
interval = setInterval( =>
if xhr.readyState > 2
total = parseInt(xhr.getResponseHeader('Content-length'))
completed = parseInt(xhr.responseText.length)
percentage = (100.0 / total * completed).toFixed(2)
console.log "Completed #{percentage}%"
, 50)
xhr
complete: ->
clearInterval(interval)
success : (data) =>
alert(data)
}
</code></pre>
</div>
<p>It’s a bit painful because of the way you access the raw <code class="highlighter-rouge">xhr</code> object in jQuery 1.5, but works nicely. I then store the data in localStorage (as a bit json blob) and we’re away.</p>
Wellington.js2011-06-27T00:00:00+00:00http://bnolan.github.com/2011/06/27/wellington-js<p>I’ll be presenting at Wellington.js on the 13th of July about Backbone.js and my experiences with Capt, my javascript build tool.</p>
<p>That is all.</p>
Webkit Transform2011-06-26T00:00:00+00:00http://bnolan.github.com/2011/06/26/webkit-transform<p>I was quite taken with these <a href="http://www.minecraftforum.net/topic/37409-mcmap-isometric-renders-sspsmp-18-support/">isometric minecraft renders</a>, so I decided to dig out some of my old game code and try and build an isometric renderer…</p>
<p><img src="/images/webkit-transform.png" />
<cite>A little 3d world</cite></p>
<p>That is created using webkit transforms, the minecraft textures. The heightmap is some perlin noise that I same via getImageData. Now to add some collision detection so you can walk around the world.</p>
Disconnected Sundays2011-06-24T00:00:00+00:00http://bnolan.github.com/2011/06/24/disconnected-sundays<p>Four weeks on from my <a href="http://bennolan.com/2011/06/11/disconnected-sundays.html">original post</a>, I wanted to do a follow up about the “Day of Disconnection”.</p>
<p>The “day of disconnect” is now one of my favourite things. I look forward to it all week. It’s the day you don’t get sucked into the computer, and you have no excuses to go outside and do all the wonderful things in the world. And when you don’t feel like going out, you are <em>forced</em> to go out and find something fun to do.</p>
<p>My first Sunday was a day of drinking, but the following Sundays have been a lot more productive. Most recently, my partner and I drove up to the east cape of New Zealand in a little convertible and walked up to the easternmost lighthouse in the world. Another Sunday was spent exercising and collecting driftwood for a big bonfire out the back of the house, and I spent another Sunday doing some vigorous gardening.</p>
<p>So - I recommend you try disconnected Sunday. It’s a truly wonderful thing, and makes your Monday coffee + internet fix even better.</p>
<p>At this rate I’m going to end up making a nicely designed <code class="highlighter-rouge">sundaydisconnect.com</code> site. Hah. :)</p>
Jasmine and Backbone2011-06-14T00:00:00+00:00http://bnolan.github.com/2011/06/14/rspec-and-backbone<p>I use Jasmine to spec / test my Backbone apps. It can be a pain setting up a Javascript testing environment (unless you’re using <a href="/capt/">capt</a> in which case you get a testing environment for free - but I digress. I decided to pull out some code from an existing app of mine and document a bit of it.</p>
<p>This isn’t best practise stuff, I’m making this up as I go along, but it is the summary of several years of Javascript development on desktop and mobile.</p>
<p>Start test suite:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>describe 'places controller', ->
describe 'search view', ->
</code></pre>
</div>
<p>I like to nest my suites so that any errors point to the correct controller and view.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>beforeEach ->
window.app = new Application
window.app.location = { latitude : -45, longitude : 170}
</code></pre>
</div>
<p>My current app has a bit of an awful hack in that it uses a global ‘app’ instance that wraps up all the application functionality. It is easy to work with, but I don’t like it for testing, since your tests inevitably have side effects. But by recreating a new ‘app’ instance for each test you help defeat some of those side effects.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>it 'should handle the truth', ->
expect(true).toBeTruthy()
it 'should exist', ->
expect(SearchView).toBeTruthy()
</code></pre>
</div>
<p>Some sanity checks to make sure the view we’re testing exists, there are no syntax or dependency problems.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>it 'should instantiate', ->
x = new SearchView { collection : new PlaceCollection }
expect(x instanceof SearchView).toBeTruthy()
expect(x instanceof Backbone.View).toBeTruthy()
</code></pre>
</div>
<p>Now we start to do something useful, by instantiating the SearchView. Notice that the view can be instantiated outside of the app, and without a specified element. Backbone will create it’s own div element for the view.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>it 'should have render method', ->
x = new SearchView { collection : new PlaceCollection }
x.render()
expect(x.el).toBeTruthy()
expect(x.el.find('input').is('input')).toBeTruthy()
</code></pre>
</div>
<p>Now we can start to do some ‘functional’ testing and make sure that the view renders up an input inside my view. This is where you can start to do some poke and prod testing, and make sure the result is as you expect it. Note that I’m running my test suite inside safari, but which something like jsdom, you could run this test suite entirely inside node.js.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>it 'should render nearby', ->
x = new SearchView { collection : new PlaceCollection }
x.collection.refresh $fixtures.placesNearby
x.collection.status = 200
x.render()
expect(x.el.html()).toMatch /harvard square/i
</code></pre>
</div>
<p>I use capt to populate my $fixtures object, so I can edit my fixture data in yaml or json, and then poke it into the view and check it renders correctly. This is the weakest selector I could use. Something like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>expect(x.$('a:first')).toMatch /harvard square/i
</code></pre>
</div>
<p>Might be better, or using the :content selector.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>it 'should render searching', ->
x = new SearchView { collection : new PlaceCollection }
x.render()
x.el.find('input').val("Kirkla").trigger('keyup')
expect(x.query).toEqual("Kirkla")
expect(x.el.html()).toMatch /loading.../i
</code></pre>
</div>
<p>You can also trigger keyUp / keyDown and other events using jQuery <a href="http://api.jquery.com/category/events/event-object/">event object</a>. Note that in my case, this will trigger an ajax call to the server, which must be mocked out on the client so that the ajax call returns immediately and fires the success callback. I use the built in mocking support (spyOn) that comes with Jasmine for mocking.</p>
Travelling2011-06-13T00:00:00+00:00http://bnolan.github.com/2011/06/13/travelling<p>I’m writing this post from the above the wing of a 18 seater twinprop plane, with the sun going down to my left, setting over mount ruapehu, which looks to be getting a good coating of powder for this years ski season.</p>
<p>Last month I decided to put my German trip on indefinite hiatus, and I’m going to be honest, it was a hard decision. But now four weeks later and I’m starting to enjoy this life. I’m living in a lovely beach house with some good guys. Travelling to Wellington city regularly to get a fix of friends, tech and cafe culture. I’ve been working with a great group of hackers based in Soho, London - apart from the early it’s an excellent way to work.</p>
<p>The next few months aren’t planned out, but I know I’ll be in Wellington for some of July, house-sitting my sisters lovely new house in Newtown, and I expect there’ll be a mission down to the Southern Island - assuming it’s still there after this newest round of earthquakes :(. Finally, my accountant reviewed my situation and helped me set a goal for the next two years, so maybe I’ll be about to get a piece of dirt to call my own in a year or two.</p>
<p>Anyway - hope everyone is healthy and happy and not persuing their dreams at the cost of their friendships.</p>
Disconnected Sundays2011-06-11T00:00:00+00:00http://bnolan.github.com/2011/06/11/disconnected-sundays<p>Nick at <a href="http://rankers.co.nz/respect/">Rankers</a> turned me onto a new movement. It’s disconnected Sundays. I’ve raised it with Sam Minnee from <a href="http://silverstripe.com/">Silverstripe</a>. The goal is to not do any work, or to use the web on Sundays.</p>
<p>Basically, avoid using your computer or iphone at all costs. It freaked me out the first Sunday - what am I going to do to entertain myself? Obviously you can just read books. Or if you feel the need to work on your project you can draw your UI plans with pen and paper - but whatever you do, as long as you’re away from the computer, it’s guaranteed to be a good day, and you’ll come back extra charged up for Monday.</p>
<p>Anyway - so here’s to being unavailable on Sundays and spending it with friends, or with your mountain bike, or working on the garden, maybe jamming in the back yard with some guitars and a drumkit.</p>
Backbone - get or fetch2011-06-10T00:00:00+00:00http://bnolan.github.com/2011/06/10/backbone-get-or-fetch<p>A common mistake people make in Backbone apps, is writing a view that assumes that data exists in a collection, so that when the user reloads the page, the collection is empty and the view won’t render.</p>
<p>For example, you have a Posts collection and a controller:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class Posts extends Backbone.collection
# ...
class PostsController extends Backbone.Controller
routes :
"" : "index"
"topics/:id" : "show"
index: ->
Posts.refresh {
success: ->
new PostsView { el : $(body), collection : Posts }
}
show: (id) ->
new PostsShowView { el : $(body), model : Posts.get(id) }
</code></pre>
</div>
<p>The problem with this, is that when you reload the page and you access the index view, it fetches the data with refresh(), then waits until the posts collection has been fetched and renders the view. (This is a bit of a contrived example, but it demonstrates the behaviour).</p>
<p>After the collection has been populated, you click on a post, and it redirects to #posts/12345, a post id that is in the collection. All good.</p>
<p>The problem is when you hit command-r and reload <code class="highlighter-rouge">localhost:3000/#posts/12345</code>. The show view will be called straight away, and the posts collection was never populated.</p>
<h1 id="getorfetch">GetOrFetch</h1>
<p>I like to fix this with a little helper called getOrFetch:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class Posts extends Backbone.collection
url: "/posts"
getOrFetch: (id) ->
if @get(id)
post = @get id
else
post = new Post { id : id }
@add post
post.fetch()
post
</code></pre>
</div>
<p>Let’s be honest, this isn’t magic, but it means you can replace your show action with:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>show: (id) ->
new PostsShowView { el : $(body), model : Posts.getOrFetch(id) }
</code></pre>
</div>
<p>Which means that no matter how you get to the show action, your model will be loaded from the server, or just use the local version.</p>
<h1 id="status">Status</h1>
<p>I’ve taken to using http status codes to represent what the status of a model is. For example a model that has been initialized but not loaded is status 0, if it is being loaded it is status 100, if it is loaded it is 200, if there was an error or the id was not found, 500 or 404, etc. This means that your view can have some code like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><% if model.status <= 200: %>
Loading...
<% else: %>
<%= model.escape 'name' %>
<% end %>
</code></pre>
</div>
<p>Doing this in the mobile app I’m working on at the moment gives a nice ios-ish interface. To make your model status update:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>getOrFetch: (id) ->
if @get(id)
post = @get id
else
post = new Post { id : id }
@add post
post.fetch {
success : ->
post.status = 200
post.trigger 'changed'
error : (e) ->
post.status = 500
post.trigger 'changed'
}
post
</code></pre>
</div>
Sunset view2011-06-06T00:00:00+00:00http://bnolan.github.com/2011/06/06/sunset-view<p>I’ve moved out to the east coast of New Zealand to work onsite with a <a href="http://rankers.co.nz/">client</a>. I’ve been gardening, programming, brainstorming, prototyping, surfing, swimming and mountain biking. It’s been a nice time.</p>
<p><img src="/images/view-out-window.jpg" />
<cite>View out the window from the office</cite></p>
<p>I’ll probably be in Wellington for the month of July, but apart from that I’m free to move around. If you have any interesting Javascript work on later in the year, I’d be keen to do a 2-3 month stint onsite. Anywhere in the world is fine.</p>
Offline Mapping in HTML52011-06-03T00:00:00+00:00http://bnolan.github.com/2011/06/03/offline-mapping<p>I’ve been doing some research and development around offline mapping for a client of mine. Here’s my notes on the current state of technology.</p>
<h1 id="tile-based-offline-mapping">Tile based offline mapping</h1>
<p>The easiest way to do offline mapping is to use something like openlayers and store tiles offline. This is kind of gross at first glance, but in practise it works well. It’s very simple, the technology is mature, and above a certain zoom level, it can actually be smaller to store the raster data than the underlying unprocessed vector data. However, it clearly doesn’t work for high zoom levels.</p>
<p>To map an area the size of New Zealand using offline tile caching to zoom level 15, a naive implementation would take about 25 gigabytes of storage. With a little development you could probably get it down to 1-2gb, and with some serious development (adaptive zoom levels based on feature detection in the underlying vector data) you could maybe get to a ~200mb download, but it’s clearly not a trivial download.</p>
<p>However - this isn’t such a big problem if you’re doing a small area like a city or a town, So - assuming you don’t need ultra-high zoom levels, and you can solve the issues of download size, what technology exists to do offline mapping?</p>
<h1 id="delivering-offline-maps">Delivering offline maps</h1>
<p>The team at <a href="http://mapbox.com/">mapbox</a> have created a suite of tools for offline mapping. Maps on a stick is an interesting tool that uses openlayers, firefox and html5 to create an offline mapping tool. It is written in javascript, and requires that you download the map tiles and store them on your local filesystem. The downside to this is that copying over 800,000 5kB tiles takes approximately forever. To solve this issue on their ipad app, Mapbox created the <a href="http://mbtiles.org/">mbtiles</a> format, which is involves putting all the maptiles into a sqlite database that modern filesystems can move around much easier.</p>
<p>Sadly, maps on a stick doesn’t support mbtiles, so I looked into what would be involved in adding mbtiles support to maps on a stick. Basically, it boils down to creating a firefox xpcom component (in javascript) that uses the firefox sqlite interface to expose the tiles to openlayers. It’s doable, but fiddly.</p>
<p>On the plus side, you could then wrap up maps on a stick, your mbtiles components and xulrunner to create an installable offline mapping tool. From my experiments with xulrunner and reading through the xpcom documentation, this is non-trivial.</p>
<h1 id="chrome-store">Chrome store?</h1>
<p>You could just shortcut all the xulrunner / xpcom stuff and do your offline apps using the chrome app store, or safari html5 offline support. I prototyped this and made a phonegap-powered ipad app that downloaded tiles from <a href="http://cloudmade.com/">cloudmade</a> and stored them in the html5 web database store. The downside to this is downloading the tiles is sloooww, and although you can do multiple requests at once using multiple subdomains, you still get slower throughput than if all the tiles were globbed together into one sqlite database.</p>
<h1 id="vector-data-and-c">Vector data and C++</h1>
<p>The best solution, if you were keen to do some serious development, would be to bust out your compiler and package up mapnik or another renderer to act as an xpcom component for firefox. You could still serve tiles, so you could use openlayers (or tile5 or leaflet), and you would have infinite zoom, nice small vector data (the OSM dataset for New Zealand is about 25mB) and you would win all the technical brownie points.</p>
<p>Down this path also lies writing your UI in QT (ala Google Earth) or WX instead of using Firefox, so you could create a completely custom mapping tool.</p>
<h1 id="analysis">Analysis</h1>
<p>I haven’t finalized my decision yet, but it seems to me that there are no natural inflection points on the cost/benefit graph for offline mapping. If anything, I might lean towards xulrunner + something like “maps on a stick”, but it’s neither here nor there.</p>
<p>Basically, there are no good offline mapping solutions for netbooks / laptops, so maybe there’s a market niche there.</p>
Hutch - tile-based game engine2011-05-30T00:00:00+00:00http://bnolan.github.com/2011/05/30/hutch<p>I’ve opensourced the repository holding my <a href="https://github.com/bnolan/island">tile-based game engine</a>. I worked pretty solidly on it for about two months (and put a bunch of videos up at <a href="http://reddit.com/r/hutch/">reddit.com/r/hutch/</a>), but it’s become a back-burnered project, so I’ve donated it to the html5 gamedev community.</p>
<p>The game was written before i developed capt, so all the code is located in <a href="https://github.com/bnolan/island/tree/master/public/scripts">public/scripts/</a> and hosted by rails. It’s designed to work with the <a href="http://www.lostgarden.com/2007/05/cutegod-prototyping-challenge.html">cutegod</a> tiles, and it’s probably a real PITA to get the thing set-up and running. I’d like to extract the coffeescript source and convert it into a capt project, so that it’d be easier for people to get the source up and running (and also start adding specs for everything), but don’t hold your breath on that one.</p>
<h1 id="orthographic-3d">Orthographic 3d</h1>
<p>The projection is orthographic, with y and z sharing the same axis (depth is faked using the z-index). The engine is designed in 3d, so positions are expressed as <a href="https://github.com/bnolan/island/blob/master/public/scripts/models/vector.coffee">3-vectors</a>. Collision detection is done in the <a href="https://github.com/bnolan/island/blob/master/public/scripts/models/player.coffee">player model</a> - if I started work on this again, I’d probably go for an architecture where classes have components, instead of a o-o big hierachy tree.</p>
<h1 id="multiplayer">Multiplayer</h1>
<p>There is some <a href="https://github.com/bnolan/island/blob/master/lib/socket_connection.rb">websocket multiplayer</a> support in the game. The ruby-based server was basically just to prototype and experiment with realtime multiplayer in the browser. In practise I’d probably use a node-based server so that you didn’t have to duplicate code from coffeescript to ruby (for example the vector class, player class, etc, etc).</p>
<h1 id="future">Future</h1>
<p>I’m still very interested in multiplayer games in the browser. I’d love to build something like minecraft, second life or terrarium, where it’s basically a sandbox for people to build worlds and gadgets for other people to explore - but it’s obviously a massive project, and if I did more work on html5-gaming, it’d probably be as part of a team.</p>
Capt Tutorial2011-05-27T00:00:00+00:00http://bnolan.github.com/2011/05/27/capt-tutorial<p>I got halfway through a tutorial on <a href="/capt/">capt</a>, my build tool for single page javascript applications using coffeescript, backbone.js, eco and less. The tutorial is available <a href="/capt/tutorial.html">here</a>, but it kind of stops halfway down. Feel free to fork the tutorial on github and submit patches, otherwise I’ll finish it up when I have time.</p>
Wellywood2011-05-26T00:00:00+00:00http://bnolan.github.com/2011/05/26/wellywood<p>I joined the Greens party in the afternoon, met the leadership in the evening and then was interviewed by the film crew at the backbencher - because I support the Wellywood sign. Video below. It was a fun night.</p>
<iframe width="340" height="240" src="http://www.youtube.com/embed/UFlYBjdTHCM" frameborder="0" allowfullscreen="true"> </iframe>
Breakfast or lunch?2011-05-25T00:00:00+00:00http://bnolan.github.com/2011/05/25/breakfast-or-lunch<p>I give you a time of day, you give me a name for the appropriate meal. I was trying to work out how to do this with a case / switch statement - then came across this syntax which is kinda nice:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def time_of_day_to_meal_name(t = Time.now)
{
0..5 => 'Midnight snack',
6..10 => 'Breakfast',
11..14 => 'Lunch',
15..16 => 'Afternoon tea',
17..21 => 'Dinner',
22..24 => 'Late night snack'
}.find { |k,v| k.include? t.hour}.last
end
</code></pre>
</div>
<p>What would you like for <a href="http://bonapp.com/">lunch</a>?</p>
Offline maps2011-05-23T00:00:00+00:00http://bnolan.github.com/2011/05/23/offline-maps<p>I’ve recently had a project morph from an online html5 mapping site, into an application that needs to run offline on netbooks and macbooks. So I’m investigating taking an openlayers, backbone and coffeescript application and packaging it up with Firefox webrunner (or <a href="https://addons.mozilla.org/en-us/firefox/addon/mozilla-labs-prism/">Prism</a>) as an installable package for Windows and os x.</p>
<p>Doing installable software isn’t new territory to me. At <a href="http://indigorenderer.com/">Indigo</a> we built installable packages for Windows, OS X and Linux - so I’m pretty happy to work my way through <a href="http://www.jrsoftware.org/isinfo.php">innosetup</a> or <a href="http://nsis.sourceforge.net/Main_Page">nsis</a>, and the os x packagemaker. For the map tiles, I might try and adapt the <a href="http://mbtiles.org/">mbtiles</a> specification from the guys at mapbox. This will be a problem, as the database will have to be jerry rigged into place where Firefox can see it. I’m avoiding doing any binary firefox extensions for this project, although if things go well, there is the posibility of storing vector data locally and creating a mapnik build that operates as a firefox extension.</p>
<p>Data will be stored locally using backbone and sqlite, with regular online syncing to the official point-of-interest repository. So far we only have the html5 protoype done, so there is a lot of ground to cover - but I’ll try and keep you all updated on how things go.</p>
Why Brisbane is the best startup town2011-05-18T00:00:00+00:00http://bnolan.github.com/2011/05/18/why-brisbane-might-be<p>This post comes as a surprise to me. When I planned to visit Brisbane for 8 weeks to spend some time with my girlfriends parents, I didn’t expect the town would grow on me so much. My previous opinion of Brissie have been is as a cultural wasteland of chromed-up bars and old holdens.</p>
<p>How wrong I was. I’ve come to really like brisbane, and I think it may be one of the best towns to do a startup in Australia / New Zealand.</p>
<h1 id="great-technology-scene">Great technology scene</h1>
<p>There is a great network of javascript, rails and other technology groups. They meet regularly, have some exceptionally strong developers (tile5, sproutcore and several other good opensource projects are maintained from brisbane). If you’re boot strapping, there is good demand for web developers working on modern projects, so you can make a good packet consulting.</p>
<h1 id="support-for-startups">Support for startups</h1>
<p>The complex of buildings along southbank are invaluable for people working from home or without a fixed office. The edge is a beautifully designed digital workshop where you can grab a couch and free wifi for a few hours. Next door is the national library which has hundreds of modern and beautifully designed workstations that are free. Both universities have modern study halls where you just walk in and grab a desk. Mobile broadband is much cheaper in Australia than in New Zealand, so you can work from anywhere. The weather is better than Wellington or Melbourne so you can easily work outdoors wherever you can find shade and somewhere comfy. You’re just over an hour from Sydney by plane, so you can head down and pitch the collection of investors there, without having to live in Sydney.</p>
<h1 id="low-cost-of-living">Low cost of living</h1>
<p>This is a comparative thing, but compared to Sydney or Melbourne (the only other two cities that could pay comparably to Brisbane) the cost of living here is much lower. A room in a cool part of town (within easy biking distance of the city) will cost you about $150 / week, which is less than half of what you’d pay in Sydney. Food and transport is cheaper than New Zealand (better public transport), and houses are better designed for the weather (compared to Wellington / Dunedin) so your energy costs will be lower.</p>
<h1 id="great-standard-of-living">Great standard of living</h1>
<p>This is the bit that really rubbed off on me. When you first get here it’s all chromed up bars, bouncers and awful franchised cafes. But Brisbane has some gems, and it has enough gems that once you link them all up you can have a really good quality life here. It’s not as epic a cafe scene as Wellington, and I miss some of the fast food in Melbourne (lord of the fries!), but Brisbane does really well.</p>
<h1 id="the-easy-long-term-view">The easy long term view</h1>
<p>There are as many types of startups as there are types of people, but in the kind of “startup” I find myself working on (part time projects funded by occasional consulting gigs, where the project might turn into a business over time), Brisbane is a really good match. It’s basically a overgrown country town, with all the advantages of affordability and relaxedness, combined with excellent modern infrastructure and a burgeoning creative sector.</p>
Backbone Gotchas2011-05-17T00:00:00+00:00http://bnolan.github.com/2011/05/17/backbone-gotchas<p>If you’re using Backbone and reusing the same div for all of your views (in a single-page iphone app for example), make sure you call .unbind() on the div when you call destroy the view, backbone doesn’t do it for you, and if you don’t unbind the div, you’ll end up with a whole bunch of events (which are using delegate so they hang around) being trigger at strange and inopportune times.</p>
Designing without thinking2011-05-16T00:00:00+00:00http://bnolan.github.com/2011/05/16/designing-without-thinking<p>When you’re working on a hard project where the functionality is spread across several files and modules, and you need to do some rearchitecture. I’ll often sit down, work out how I want it to work, then zone out and just refactor without reading the code on the page. Old-school lispers are famous for this, they can work out when to refactor and where the code is unbalanced by looking at the <em>shape of the code</em>, due to all the parentheses. I do a similair thing, relying on syntax highlighting and compile-on-save to point out when I’ve broken something.</p>
<p>The second case is when you’re prototyping a project, and you have to force yourself to not overthink things, to prevent becoming an ‘architecture astronaut’ (abstracting everything away into a GenericFactoryFactoryGenereratorCreator), and instead just deciding ‘I want to to X, so I’ll copy this code here and just search / replace until everything works’. I’ve been doing a bunch of this lately with a backbone mobile project I’m working on. I’ve got about a dozen lines of code that I’m copying and pasting into every model I create. I <em>should</em> create a new base class, and inherit from that, instead of cargo-culting (which is error prone and tedious), but I’m not going to do that until copying/pasting becomes unbearably tedious. By refactoring prematurely, I’m bound to make annoying assumptions that I’ll end having to code around anyway. So don’t overthink it and just copy / paste away until the extraction to make is painfully clear.</p>
<p>Obviously both of these techniques only work in a well tested environment, and I can put aside some time the next day to go over what you did and identify the bits that were left out or done badly.</p>
<p>Oh - and there is obviously that third type of coding without thinking, which is when you need to get something done that you really don’t want to, so you grab a few beers and create variables named <code class="highlighter-rouge">the_variable_which_is_a_number_and_gets_incremented</code>.</p>
Proof of work in Javascript2011-04-28T00:00:00+00:00http://bnolan.github.com/2011/04/28/proof-of-work-in-js<p><a href="http://www.thefengs.com/wuchang/work/puzzles/globalinternet08_kapow.pdf">Proof of work</a> is the idea of getting a client to do some computational work that is expensive to perform, but cheap to verify.</p>
<p>For example - if you send a client a random number, and then ask the client to keep increasing that number until the md5 hash of the number starts with four zeroes, it will take on average 2^16 attempts to find the matching number. Then the client sends the number back to the server and the server verifies that the hash starts with 0000…</p>
<p>The idea is to make the computation expensive enough that it makes it less worthwhile for spammers to attack your system. It’s a nice alternative to a captcha because it doesn’t require any user intervention.</p>
<h1 id="example-proof-of-work-solver">Example proof-of-work solver</h1>
<div class="highlighter-rouge"><pre class="highlight"><code>var x = parseInt(Math.random() * 0xFFFFFFFF),
doWork = function(){
var count = 100, y, h;
while(--count > 0){
x += 1;
y = x.toString();
h = hex_md5(y);
if(h.match(/^0000/)){
console.log("Solved... " + x);
return;
}
}
setTimeout(doWork, 25)
};
doWork();
</code></pre>
</div>
<p>This code generates a random number, then increases it until a suitable hash is found. The work unit is broken up into 100 tests at a time, with a timeout between subsequent work so that the browser doesnt warn about script timeouts. This runs fine on a modern browser, but I imagine it would nail ie6. For production code you’d probably want to profile the doWork function and adapt the work unit size to keep the browser responsive.</p>
<h1 id="native-solvers">Native solvers</h1>
<p>One of the obvious downsides is that any javascript implementation of a hashing function is going to be much slower than a native implementation, so if a spammer wrote a native solver for your proof-of-work, they would be able to trivially solve your work requests.</p>
<p>For example - firefox takes about 8 seconds to solve my proof of work prototype, firefox takes about 6 seconds, and ruby can solve it in 0.23 seconds.</p>
<h1 id="alternative-work-algorithms">Alternative work algorithms</h1>
<p>Many hashes is a well known and provably computationally difficult proof-of-work technique, but given the problem that a browser will never be able to solve them as fast as a modern GPU, it would be worth investigating other work that a browser could solve that could not be solved faster by any other method, for example calculating the size of an HTML layout.</p>
Capt 0.5.32011-04-21T00:00:00+00:00http://bnolan.github.com/2011/04/21/capt-0.5.3<p>I’ve just pushed <a href="/capt/">capt</a> 0.5.3. Code generation works properly now <code class="highlighter-rouge">generate controller</code> and <code class="highlighter-rouge">generate model</code>, and the capt watch and capt server commands actually work (I was a bit premature announcing on Tuesday). I’m using this on two projects now and it has been a really nice toolchain.</p>
<p>Autogeneration of spec classes is good for maintainability too. I kinda wish I had been BDD-ing <a href="https://github.com/buddycloud/channel-webclient">Buddycloud</a> at the moment. That’s some complicated stuff.</p>
Wanted - A local startup2011-04-21T00:00:00+00:00http://bnolan.github.com/2011/04/21/a-local-startup<p>I’ve worked on “local” for a really long time, it’s something that’s got under my skin. Whether it’s creating mapping engines, url hierarchy for places or analyzing aggregate user trends, I just really love it. I started a startup in 2006 doing local. We almost got hire-acquired up by Google, but didn’t, long story.</p>
<p>Are there any early stage local startups out there?</p>
<p>Like one or two technical people that are working on something cool and want another hacker on board? Something part time would be a great start for me, I’m happy to toodle along for a while on a cool idea while we work out to get traction.</p>
<p>Things I’m interested in:</p>
<h1 id="quora-for-local">Quora for local</h1>
<p>People ask ‘where’s a good place to get a beer in wellington?’ and you reply “[Hashigo Zake] and [The Hop Garden]” and the UI pops up little boxes saying ‘Did you mean Hashigo Zake at 25 Taranaki Streeet’? And you click yes and it turns your square brackets into a link.</p>
<h1 id="reddit-for-local">Reddit for local</h1>
<p>People can click on places in a city (reverse geocode to buildings and natural features using a combination of google local search and openstreetmap gazetter) and each place has a discussion board. Going to a city level shows all the conversations going on in the city, sorted by recentness. Build a url scheme like <code class="highlighter-rouge">/paris/la-cite/notre-dame</code>. I’ve written a prototype of this.</p>
<h1 id="delicious-for-local">Delicious for local</h1>
<p>Write a bookmarklet that lets you bookmark a webpage, and it hunts out any geodata in the page and saves it to a map for you. You can use this to plan your trips by saving all the interesting looking webpages and then accessing the list of places offline on your iphone. I’ve written this bookmarklet already.</p>
<h1 id="greplin-for-local">Greplin for local</h1>
<p>The user signs in using oauth to foursquare, google my maps, facebook, gowalla and twitter. Index all the places the user has been and all the places their friends go. Use this to generate a ‘personal black book’ of places to go while travelling the world. So you go to berlin, fire up ‘GeoGreplin’ and it shows you all the places your friends want to visit / have checked in to / have saved to a map.</p>
<h1 id="twitter-local-browser">Twitter local browser</h1>
<p>Suck down the twitter geo-firehose and plot all the tweets onto maps. Break the tweets down into cities and suburbs. Aggregate the tweets together (using some kind of <a href="http://bennolan.com/2010/09/10/contour-maps.html">2d histogram</a>) to show where the interesting stuff is in your city. People can discover people nearby and see what’s going on. I’ve built a prototype for this too.</p>
<p>Anyway - that’s the interesting ideas. Anyone working on stuff like this? Or any angel list types want to fund me to find a cofounder and build one of these?</p>
There comes a time in every project...2011-04-20T00:00:00+00:00http://bnolan.github.com/2011/04/20/there-comes-a-time<p>There comes a time in every project - after days of prototyping, design, tweaking, rigorous speccing and bdd, after several deployments to staging and a/b testing with beta users, after polls and roundtables and whiteboarding - that you have to admit that the project isn’t any fun to use.</p>
<p>And that’s the opportunity to sit back and reimagine it, then make some small (usually not huge) tweaks until it starts to come together again.</p>
Channel Activity2011-04-20T00:00:00+00:00http://bnolan.github.com/2011/04/20/channel-activity<p>It’s hard to discover new activity in Buddycloud, time to fix that.</p>
<p><img src="/images/poke/channel-list.png" />
<cite>The channel list in buddycloud</cite></p>
<p>Log into diaspora-x and there’s no indication of what activity is happening on your channels. Simon said to put the channel list on the left and bubble activity to the top. Let’s try.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>span{
display: inline-block;
height: 20px;
width: 20px;
text-align: center;
font-size: 0.8em;
background: #999;
color: white;
border-radius: 10px;
}
</code></pre>
</div>
<p>Okay - add a litte badge showing the number of posts, and reorder the channel list by number of posts.</p>
<p><img src="/images/poke/channel-list-imp.png" />
<cite>A sorted list of channels with new activity</cite></p>
<p>Nice.</p>
Diaspora-x ui work2011-04-19T00:00:00+00:00http://bnolan.github.com/2011/04/19/diaspora-x-ui<p>I’ve been doing some work on buddycloud / diaspora-x user interface. Screenshots follow:</p>
<p><img src="/images/sam-friends.png" />
<cite>You have to be friends</cite></p>
<p>I have to admit that the buddycloud ui looks a lot nicer, with blues and colors, but I’ll be sticking with the grey and black ui that diaspora-x features.</p>
<p><img src="/images/the-ministry.png" />
<cite>The ministry of interesting posts</cite></p>
<p>I’m getting ready to deploy the latest diaspora-x.</p>
Capt is in npm2011-04-18T00:00:00+00:00http://bnolan.github.com/2011/04/18/capt-is-in-npm<p>I’ve resumed work on <a href="http://bennolan.com/capt/">capt</a>, since my <a href="http://diaspora-x.com/">diaspora-x</a> codebase relies on capt, and although I expect <a href="http://brunchwithcoffee.com/">brunch</a> or <a href="http://cincojs.com/">cinco</a> will take over as the build tool of choice for coffeescript / backbone apps - capt works for me now.</p>
<p><img src="/images/capt.png" />
<cite>Logo indicates far more polish than the project has.</cite></p>
<p>Capt is a tool for development and deploying javascript single page apps. It takes care of including all your javascripts in the right order, and compiling your static html pages. It has some support for jasmine, coffeescript and less. It’s very creaky, but my collaborators on the diaspora-x / Buddycloud web client needed to be able to use it, so I’ve packaged up capt as an npm.</p>
<p>So you can install capt with:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>npm install capt
</code></pre>
</div>
<p>Once it’s installed, create a new app and serve it using the built in server.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>capt new myproject
cd myproject
capt server
open http://localhost:4000/
</code></pre>
</div>
<p>And enjoy the capt goodness. I’ve put up a <a href="http://bennolan.com/capt/">basic webpage</a> explaining how to use capt. I imagine this release is pretty broken, it’s only really intended for diaspora-x development, but give it a go. I’ve been doing small commits on capt for a while now, so it’ll eventually turn into a decent tool I hope.</p>
Coffeescript supersedes Rails2011-04-15T00:00:00+00:00http://bnolan.github.com/2011/04/15/coffeescript-supersedes-rails<p>This is a bit of a <a href="https://github.com/rails/rails/compare/9333ca7...23aa7da">contentious post</a> (that commit is comedy gold) but I think people talking about rails using coffeescript as the new default client-side language are missing the point that coffeescript doesn’t complement rails, it supersedes it.</p>
<p>I come from a biased position, since I’m known for front end development, but I’ve been doing rails since version 0.6, and most of my contract gigs have a ruby on rails component. Rails is a great tool for building middleware, and when it was invented it was a perfect solution.</p>
<h1 id="2006">2006</h1>
<p>Common technology was MySQL and PHP. Front end development was pretty low-impact, mostly you generated html on the server and sent it to the client, sometimes you sent html fragment back and forth for some ajax-interactivity.</p>
<p>Into this environment, rails was amazing. It was a great framework for building apps, gave you conventions on how to structure your app, forms and data models. And then rails came out with the ajax helpers and people went crazy for it. Over time rails added excellent json support and people started using it to create pure json api sites (like twitter does these days), but fastforward to 2011…</p>
<h1 id="2011">2011</h1>
<p>We now have pure json databases (couchdb, mongodb), localStorage and thousands of public APIs. And any truly modern app will probably be written in pure javascript (and rendering their static html by a node.js instance for disabled browsers). Into this environment, Rails makes less sense. If you’re building a facebook add-in, you can probably do it entirely using FBJS. If you’re using the foursquare api you can do that in the client, same with twitter. If you need to do an app that stores tonnes of data - get a free couchdb instance from couchone. If you need “web scale” - you can use amazon simple db.</p>
<p>Rails doesn’t need to feature in any of this. And although rails will live on forever (like php does), and it’ll probably adapt to it’s new role as an authentication and permissions layer between the client and the database, I would be surprised if a better solution doesn’t come along. It could be a more mature CouchDB authentication / permissioning framework, or Postgres might release a json interface and people go back to writing stored procedures to authenticate / secure requests.</p>
<h1 id="middleware-is-becoming-less-important">Middleware is becoming less important</h1>
<p>I’m sure its just a cyclical thing, we go back and forth between thin client to thick client architectures, but it seems to me that rails might be on the downswing, to be replaced by some hot new tech that takes less work to create pure javascript apps.</p>
<p>Note that I’m not saying node.js is going to replace rails. I think there’s less need for middleware these days.</p>
Planning in the open2011-04-14T00:00:00+00:00http://bnolan.github.com/2011/04/14/planning-in-the-open<p>I’ve been designing today. The current <a href="http://buddycloud.com/">Buddycloud</a> site needs a refresh - so I busted out Jekyll (the tool I use to generate bennolan.com), some css3 and the blueprint css framework.</p>
<p><img src="/images/bc-website-shot.png" />
<cite>The in development buddycloud site</cite></p>
<p>It came together quite nicely, and because we’re hosting the repository for the site on github - you too can go <a href="http://buddycloud.github.com/">visit the site</a> - even though it’s unfinished.</p>
<p>Which brings a dilemma. Do we make the repository private - remove the hosted instance and only require people working with Buddycloud to view the in-development site? There’s sure some interesting / awkward stuff in there - a lot of lorem ipsum text and some rough guesses of price points and a business model.</p>
<p>Well - to be honest, what kills most small businesses (which is what Buddycloud is at the moment) is obscurity or apathy, not giving interested parties too much insight into your business. So we’ll leave it up for now, and then everyone can see the site when it’s ready for ‘public’ deployment.</p>
Localstorage for speed2011-04-13T00:00:00+00:00http://bnolan.github.com/2011/04/13/localstorage-for-speed<p>The Buddycloud web client is <a href="https://github.com/buddycloud/channel-webclient">available on github</a>. I started with the Diaspora-x codebase, and have been backporting in features from the codebase that was started by <a href="http://astro.soup.io/">Stefan Maka</a> (another Buddycloud contributor).</p>
<p><img src="/images/channelinfo.png" />
<cite>Channel metadata display</cite></p>
<h1 id="using-localstorage-to-accelerate-load-time">Using localStorage to accelerate load time</h1>
<p>When you fire up <a href="http://diaspora-x.com/">diaspora-x</a>, it takes about 15 seconds to finish loading the #home page. Obviously this isn’t going to play in a world where page loads should take <a href="http://www.useit.com/alertbox/response-times.html">less than 1000ms</a>. Ideally I’d like the page to load in about 100ms. Assuming we can get everything cached locally, that gives a pretty generous time for the browser to cache and run the javascript payload.</p>
<p>The first part of the optimization was to prevent unnecessary requests to the XMPP server. Once you’ve requested all a users posts, there’s no need to re-request the same posts. We can use <a href="http://xmpp.org/extensions/xep-0059.html">Result Set Management</a> to only ask for posts newer than the last post we recieved, and then we can use <a href="https://github.com/jeromegn/Backbone.localStorage">backbone-localStorage</a> to save the posts in localStorage in the current browser.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class ChannelCollection extends Backbone.Collection
model: Channel
initialize: ->
@localStorage = new Store("ChannelCollection")
</code></pre>
</div>
<p>And then before we start the app…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Channels = new ChannelCollection
Channels.fetch()
</code></pre>
</div>
<p>Then remember to clear localStorage when the user logs out.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>localStorage.clear()
</code></pre>
</div>
Distributed Social Networking2011-04-12T00:00:00+00:00http://bnolan.github.com/2011/04/12/distributed-social-networking<p>Normally when a client rings up and wants you to build a replacement for Facebook, you smile politely and say that you’re full up for the next 18 months.</p>
<p>But, <a href="http://www.facebook.com/simon.tennant">Simon</a> from <a href="http://buddycloud.com/">Buddycloud</a> has asked me really nicely to build the front end for his distributed social network, so I said I’d give him 5 weeks and see what we can pull off.</p>
<h1 id="buddycloud">Buddycloud</h1>
<p>The buddycloud protocol is an extension for XMPP, the system that gtalk uses. Buddycloud takes the existing XMPP extensions (publish / subscribe, rosters, federation) and builds a social network around them. Buddycloud is an opensource protocol, Simon and others are trying to get the protocol in front of the xmpp standards committee.</p>
<p>There are three implementations of buddycloud-compliant servers at the moment, one for ejabberd, one for prosody, and a new one that uses node.js and plugs into any standards-compliant jabber server.</p>
<h1 id="web-front-end">Web front end</h1>
<p>Back in December, I created <a href="http://diaspora-x.com/">Diaspora-x</a>, which was a first cut at building a social network on top of XMPP. It is a pure HTML5 app, just javascript, html and css - the backend service is provided by the XMPP servers (running <a href="http://www.ejabberd.im/">ejabberd</a> and <a href="http://prosody.im/">prosody</a> at <a href="https://beta.buddycloud.com/http-bind/">beta.buddycloud.com</a>.</p>
<p>This was an interesting experiment, and showed it is possible to build a responsive social network that gets all it’s data from an xmpp network.</p>
<h1 id="twitter-facebook-and-buddycloud">Twitter, Facebook and Buddycloud</h1>
<p>It’s interesting to note that <a href="http://engineering.twitter.com/2010/09/tech-behind-new-twittercom.html">new twitter</a> uses the exact same engineering as Diaspora-x, in that when you visit the website, you’re actually loading a large javascript application. The tweets and profile data that you see is transferred afterwards by an XMLHttprequest.</p>
<p>The existing Diaspora-x client (which is the basis for the new Buddycloud web client) does the same thing, it’s a Javascript app, powered by Backbone, jQuery and Underscore.js - which instead of loading tweets from api.twitter.com, it subscribes to an XMPP channel over BOSH (XMPP over http protocol), and messages from your contacts are pushed down to the browser.</p>
<p>The current Facebook UI does a similair thing, it loads a small amount of static html when you load the page, then most of the rest of the page is constructed from Javascript running in the browser. (Facebook is a bit different because it uses their <a href="http://www.facebook.com/note.php?note_id=307069903919">Primer</a> tool that downloads html, instead of doing the templating in the browser - but similar concept).</p>
<h1 id="building-the-buddycloud-web-client">Building the Buddycloud Web Client</h1>
<p>The Buddycloud web client will be an Opensource project, so you’ll be able to watch my progress at github once the repository is up. I’ll try and blog regularly here, so watch for progress…</p>
A* in coffeescript2011-04-11T00:00:00+00:00http://bnolan.github.com/2011/04/11/astar-in-coffeescript<p>I put together this astar solver in coffeescript. It expects that nodes have a <code class="highlighter-rouge">key()</code> function that generates a unique id for the node, and an <code class="highlighter-rouge">getAdjacentNodes()</code> function that returns valid adjacent nodes.</p>
<p>It seems to work nicely.</p>
<p>Also - it’s my birthday today! Woo! :)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class AStar
constructor: ->
@openNodes = {} # List of openNodes nodes (nodes to be inspected)
@closedNodes = {} # List of closedNodes nodes (nodes we've already inspected)
# The maximum potential trip length we would consider
@maxHeuristic = Point.origin().squareDistanceTo(new Point(5, 5))
findPath: (start, destination) ->
# g = 0 #Cost from start to current node
# h = heuristic(start, destination) #Cost from current node to destination
# var f = g+h #Cost from start to destination going through the current node
start.f = @heuristic(start, destination)
# Push the start node onto the list of openNodes nodes
# openNodes.push(start)
@openNodes[start.key()] = start
#Keep going while there's nodes in our openNodes list
while @openNodes
#Find the best openNodes node (lowest f value)
#Alternately, you could simply keep the openNodes list sorted by f value lowest to highest,
#in which case you always use the first node
node = { f : Infinity }
for key, n of @openNodes
if n.f < node.f
node = n
# No nodes remain in openNodes
if node.f == Infinity
# No path could be found...
console.log "No path could be found"
return null
# console.log @closedNodes
# Check if we've reached our destination
if node.equals(destination)
path = [destination]
while (node != start) # && (node.parentKey)
node = @closedNodes[node.parentKey]
path.push node
path.reverse()
return path
# Remove the current node from our openNodes list
delete @openNodes[node.key()]
# Push it onto the closedNodes list
@closedNodes[node.key()] = node
# Expand our current node
for n in node.getAdjacentNodes() when (!@closedNodes[n.key()]) && (!@openNodes[n.key()])
# console.log(n.key())
n.f = @heuristic(n, destination)
n.parentKey = node.key()
# Prevent really long paths
if n.f < @maxHeuristic
@openNodes[n.key()] = n
# else
# @closedNodes[n.key()] = n
# An A* heurisitic must be admissible, meaning it must never overestimate the
# distance to the goal. In other words, it must either underestimate or return
# exactly the distance to the goal.
heuristic: (a, b) ->
a.position.squareDistanceTo(b.position)
</code></pre>
</div>
<p>Here are the specs so you can see that it does actually work, but the require a bunch of other classes that I’ve been working on - and I don’t have time to tidy them up for release right now:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>describe 'astar', ->
describe 'square map', ->
# Create a big square tile
map = new Map
bounds = new Bounds(Point.origin(), new Point(10,10))
for point in bounds.getPoints()
map.set(point, new Tile)
origin = map.get(Point.origin())
it "should work horizont", ->
a = new AStar
x2 = map.get(new Point(0,5))
path = a.findPath(origin,x2)
expect(path.length).toEqual 6
it "should work vertical", ->
a = new AStar
x2 = map.get(new Point(5,0))
path = a.findPath(origin,x2)
expect(path.length).toEqual 6
it "should work diagonal", ->
a = new AStar
x2 = map.get(new Point(5,5))
path = a.findPath(origin,x2)
expect(path.length).toEqual 11
path = for tile in path
tile.toString()
expect(path.join(" -> ")).toEqual("0, 0 -> 1, 0 -> 1, 1 -> 2, 1 -> 2, 2 -> 3, 2 -> 3, 3 -> 4, 3 -> 4, 4 -> 5, 4 -> 5, 5")
it "should several times", ->
x2 = map.get(new Point(5,5))
a = new AStar
path = a.findPath(origin,x2)
expect(path.length).toEqual 11
a = new AStar
path = a.findPath(x2,origin)
expect(path.length).toEqual 11
a = new AStar
path = a.findPath(origin,x2)
expect(path.length).toEqual 11
a = new AStar
path = a.findPath(x2,origin)
expect(path.length).toEqual 11
a = new AStar
path = a.findPath(origin,x2)
expect(path.length).toEqual 11
a = new AStar
path = a.findPath(x2,origin)
expect(path.length).toEqual 11
it "should work on a single step", ->
a = new AStar
x2 = map.get(new Point(1,0))
path = a.findPath(origin,x2)
expect(path.length).toEqual 2
it "should work on origin->origin", ->
a = new AStar
path = a.findPath(origin,origin)
expect(path.length).toEqual 1
it "should return empty when impossible", ->
map.set(new Point(100, 0), new Tile)
a = new AStar
x2 = map.get(new Point(100, 0))
path = a.findPath(origin,x2)
expect(path).toEqual null
</code></pre>
</div>
Against me nervous energies sessions2011-04-09T00:00:00+00:00http://bnolan.github.com/2011/04/09/nervous-energies<p>For future reference, the <a href="http://www.youtube.com/results?search_query=against+me+nervous+energies&aq=f">against me nervous energies sessions</a> are linked in this post.</p>
<p>My favourite band of all time. Make sure you also check out <a href="http://www.youtube.com/watch?v=kmUKY0JEArA">borne</a> and most importantly, <a href="http://www.youtube.com/watch?v=X9iTNNi8Gh0&playnext=1&list=PLCBE065BE058723BF">don’t lose touch</a>.</p>
Native maps speed in Javascript2011-04-08T00:00:00+00:00http://bnolan.github.com/2011/04/08/native-maps-speed-in-javascript<p>Whenever I use <a href="http://openlayers.org/">OpenLayers</a> or even <a href="http://www.tile5.org/">Tile5</a>, it annoys me that no matter how much you optimize your javascript code, your html5 map will never be as smooth and fluid as the native iphone maps app. I decided to try and fix that today.</p>
<p><a href="/experiments/nativemaps/">This demo</a> is only a proof of concept, but if you try it on an iphone (even my 1st generation iphone), you’ll see that it pans as quickly as the native client. It uses:</p>
<ul>
<li>A 60fps interval to update the map position</li>
<li>Damped map velocity movement (flicking)</li>
<li>Translate3d to use hardware acceleration</li>
<li>Suppresses touch events and user-select events to prevent the user interaction slowdown</li>
<li>Zepto.js for event management</li>
</ul>
<p>I’m discussing with Damon from tile5 to get this code put into tile5 (since writing an entire javascript mapping library isn’t on the cards for me today), but I wanted to show the performance that javascript mapping libraries should be aiming for.</p>
Smallest turing complete language2011-04-06T00:00:00+00:00http://bnolan.github.com/2011/04/06/smallest-turing-complete-language<p>I was wondering what the smallest set of opcodes would be for a turing complete language.</p>
<p><a href="http://en.wikipedia.org/wiki/Brainfuck">Brainfuck</a> seems to indicate that you need 8 opcodes:</p>
<ul>
<li>Move the data pointer left or right</li>
<li>Increment / decrement the data pointer</li>
<li>Write the data pointer to the output</li>
<li>Read input to the data pointer</li>
<li>Loop until the data pointer is zero</li>
</ul>
AI smallest program2011-04-05T00:00:00+00:00http://bnolan.github.com/2011/04/05/ai-smallest-program<p>I’ve been thinking about AI a bit lately, and a question I’ve had, is what is the smallest non trivial programming loop you could make? What number of opcodes would it be? I guess it’d have to be a program that took evolving inputs and iterated to approach an optimal solution.</p>
<p>I’m thinking of a program that has changing inputs, so that there is no one optimal solution, the best the program can do is constantly seek to a solution. It’s a thought experiment, in that that you can calculate an upper bound for how quickly a genetic program could evolve by working out the simplest non trivial program, and calculating how fast it could be run on a modern cpu.</p>
Simple English2011-04-04T00:00:00+00:00http://bnolan.github.com/2011/04/04/simple-english<p>Simple english has only 800 words, and I’ve found an english -> basic english <a href="http://www.basic-english.org/down/idp.zip">translator</a>. I think it’s a good vocabulary for a chatbot.</p>
<h1 id="hypergraphs">Hypergraphs</h1>
<p>I’ve been reading about hypergraphs, which is what OpenCog uses as a knowledge store. It’s a graph where instead of edges being between nodes, edges can have have an infinite number of nodes. You could look at it as a ‘set store’, where you have many many sets of different ‘strengths’, surrounding a fixed pool of nodes. Anyway, I’m not totally sure I understand the theory, but it seems like an interesting idea.</p>
<p>My naive implementation of a knowledge store for a chatbot, is to create nodes for terms, and then have bidirectional relationships between terms, and relationships are strengthened when the analyser is fed two terms in sequence. As far as i can tell, this is like the hypergraphs.</p>
<h1 id="metropolis-hasting">Metropolis-Hasting</h1>
<p>After working on <a href="http://indigorenderer.com/">Indigo Renderer</a> and having many conversations with Nick Chapman, I have a sense that when doing a random walk through the AI’s knowledgebase, that it’d be worth calculating a probability that the path would have been selected.</p>
<p>So for example - the AI queries it’s knowledge store for the term ‘dog’, and the knowledge store would query for a thousand random chains of related terms from dog. It would then rank those chains in terms of likelihood of those chains having been taught to the machine.</p>
<p>So - train the bot:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dog is animal
dog is pet
horse is animal
</code></pre>
</div>
<p>And the knowledge store might return:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>0.2 dog is animal
0.2 dog is pet
0.15 dog is animal is pet
0.1 dog is animal is
0.1 dog is pet is
0.1 dog is pet is dog
0.01 dog is animal is horse
</code></pre>
</div>
Brunch2011-03-23T00:00:00+00:00http://bnolan.github.com/2011/03/23/brunch<p>I’ve stopped developing capt, my javascript build tool. I’m instead contributing to <a href="http://brunchwithcoffee.com/">brunch</a>, a tool built by Jan and Thomas. It’s still early stages, but it is already ahead of where capt was. My todo list for brunch is:</p>
<ul>
<li>Add jasmine testing to clients</li>
<li>Port static html generation templates from capt</li>
<li>Add code generation for classes / models</li>
</ul>
<p>It’s an interesting project, hopefully a good alternative to the upcoming framework from @sstephenson.</p>
The golden ratio in CSS2011-03-22T00:00:00+00:00http://bnolan.github.com/2011/03/22/golden-ratio<p>I’ve been designing a new mapping interface for a product I’m releasing shortly. I’ve copied the design of the twitter app for ipad, which has multiple sliding panes. I was trying to work out the widths of each pane so that the app looked a bit more cohesive, when I decided to try out the golden ratio.</p>
<p>I’m using <a href="http://learnboost.github.com/stylus/">stylus</a> - which is an excellent css toolkit, similar to sass or less.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>golden-ratio = 1.61803399
base-width = 48px
</code></pre>
</div>
<p>The base-width is the width of the skinniest column on the left side of the page. I then use multiples of the golden-ratio and the base-width to chose the widths of the other columns.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>.modes-presenter
width base-width
.places-list-pane
width (golden-ratio ** 4 * base-width) - 40px
.place-show-presenter
width (golden-ratio ** 4 * base-width) - 40px
</code></pre>
</div>
<p>I’m not sure if it was a placebo effect - but the end result was quite pleasing to me.</p>
<p><img src="/images/golden-ratio.png" />
<cite>Early prototype with GR-sized columns</cite></p>
Back from holiday2011-03-21T00:00:00+00:00http://bnolan.github.com/2011/03/21/back-from-holiday<p>On the 1st of February, Riss and I moved out of our apartment on Mount Victoria, and into our little green van. We had a bed in the back, gas bbq, chillibin and a box of food. We spent the next 7 weeks travelling around the South and North islands of New Zealand.</p>
<p><img src="/images/P1050248.jpg" />
<cite>Totaranui, Golden Bay. Beautiful place.</cite></p>
<p>I had created a rough map of where to go using a beta version of Weheartplaces, but in the end we just travelled using the lonely planet and my ipad to look up places on the go. As it was, it turned out that a place bookmarking service would’ve been really awesome, especially in finding freedom camping sites, as they are scattered throughout the internet and hard to find.</p>
<p>(As it happens, I’m working on a solution on finding freedom camping sites as we speak).</p>
<p>Camping in New Zealand is beautiful. The doc campsites are the best places to go. They’re remote, cheap, relaxed and have the best locations. My favourite three places were Totaranui, Whananaka North and Aoraki campgrounds. I’m still working on culling and uploading all the photos from the trip - but it was a very fun time - and it’s great to be back.</p>
<p>Gear review:</p>
<p>Nissan 4wd Van - excellent. Big enough to sleep in (although we had a bit too much stuff), small enough to get into all the great little campsites. 4wd was good for rallying up.</p>
<p>iPad w/ 3G - excellent. Hugely useful, but not so capable that I could do any more than briefly reply to emails (can’t do any coding on it, which is good). Vodafone coverage is fucking awful, telecom 3g is much better.</p>
<p>Chilli bin - good. Buy some ice every few days and you’ve got a cheap, easy to use fridge. Only thing missing was it didnt have a tap at the bottom to let out the melted water, and it couldnt fit heaps of beers.</p>
<p>Gas cooker - excellent. You need a twin burner, so you can make fancy meals. We ate really well, and making coffee with the burr grinder and stovetop espresso, while sitting on a remote beach, is really awesome.</p>
Desaturated google maps2011-01-27T00:00:00+00:00http://bnolan.github.com/2011/01/27/desaturated<p>You’re building your site, you do it nice, you have regular line spacing, lots of white space, even gaps, nicely kerned typography. You took your triplet css values <code class="highlighter-rouge">#ccc</code> to <code class="highlighter-rouge">#c3c3c3</code> for that perfect shade of grey, changed all the borders from dotted to solid. Your website is maybe starting to look like the pages of a design magazine. Then you add google maps to the page.</p>
<p>It’s like smearing playdough on your furniture.</p>
<p>Try using google styled maps and desaturate the map to get something that looks more like this. Still functional, but easier on the eye and suitable to my bauhaus sensibilities.</p>
<p><img src="/images/desaturated.png" />
<cite>Desaturated, label opacity reduced</cite></p>
Fix myspace in 5 steps2011-01-25T00:00:00+00:00http://bnolan.github.com/2011/01/25/fix-myspace<p>I have a bit of a thing for trying to fix broken social networks (see <a href="/2010/12/07/diaspora-x.html">Diaspora-x</a>), so when I recieved an invite to friend someone on Myspace, it jogged my memory that the company is up for sale, so I thought I’d make my list of 5 things the new owner could to to stem the decline and maybe even compete with the #1 social network.</p>
<h1 id="become-a-flat-engineer-oriented-organisation">Become a flat engineer-oriented organisation</h1>
<p>From reading this <a href="http://www.reddit.com/r/IAmA/comments/f09rk/iama_myspace_employee_about_to_be_laid_off_today/">reddit iama</a>, it seems that the engineering teams aren’t communicating effectively. Given the glacial speed of improvements to the site, I imagine that the development process is pure corporate-bullshit, where any improvement or tweak must go up through a layer of bureaucracy before being built.</p>
<p>Instead, let engineers discuss a feature with their team lead, build it and deploy it in the same day. The same way facebook does. And make the default answer to be ‘yes do it’, not - ‘uhh lets think about that’. Companies fail because the employees can’t innovate, not because the employees fuck up.</p>
<h1 id="become-design-and-user-experience-focussed">Become design and user experience focussed</h1>
<p>The myspace user experience is poor when compared to Facebook. It’s hard to tell where you are in the site, it’s hard to find information, it’s hard to work out how to do simple tasks. The biggest things that need fixing with the UI right now:</p>
<ol>
<li>Myspace style guide. All myspace pages, from the developer guide to the front page to the sitewide UI should obey the same style guidelines for consistency.</li>
<li>Focus on page load time and browser stability. Deprecate flash, promote and develop html5 solutions for the current widgets. Don’t autoplay any media.</li>
<li>Add summaries of profile sub pages on the main profile page (show top 3 photo thumbnails, links to the top 3 tracks on the music page, create a summary field that shows the user / artists most recent activity) to encourage users to explore deeper into the profile and not just hit ‘back’.</li>
<li>Add more explorability. Make record label links go to a ‘label page’ that shows other artists. Make the ‘location’ field a hyperlink that shows a page with all public profiles located in that area.</li>
<li>Hire pixel-perfect designers. You don’t have the facebook ui template library, so make developers forward a screenshot of any new UI (be it a ‘you are logged in’ alert or a ‘changes saved’ interstitial) to your design team, and give the design team a mandate to sit in with developers and show them how the pixels should be moved around until everything lines up.</li>
</ol>
<p><img src="/images/badlayout.png" />
<cite>Galloping baseline, overlapping elements, multiple colors, inconsistent spacing, no avatar image.</cite></p>
<h1 id="become-more-open">Become more open</h1>
<p>Myspace should move their APIs over to the opengraph and do a staged deprecation of their current APIs. The focus should be on letting third party developers access the myspace data. Creative commons licences should be promoted and liberal licences should be granted to myspace data. Users should have control over their privacy, but anything that has been elected to be public should be shared freely and instantly, no more summary-only rss feeds, no obfuscated URLS, all public content should be freely and easily available.</p>
<h1 id="make-better-tools-for-bands">Make better tools for bands</h1>
<p>The tools for bands to edit their tour schedule, keep in contact with the fans or maintain their page should have a UI overhaul. A nice, simple, ajaxified design that lets them get in, update their tour and music and then get out again. The bands data should be made available through APIs if the band decides too - so that myspace becomes the central location they manage their social media presence. For example, their myspace page should have a podcast url, their tour data should be syndicated out to upcoming, last.fm, they should be able to set creative commons (or other) licences on all their media.</p>
<h1 id="start-a-dialog-with-your-community">Start a dialog with your community</h1>
<p>Most people think that Myspace is fucked. You need to start communicating on the myspace blog, often and always, and show the work that is being done to make Myspace a better place. You need to point out where Facebook fucked up and you did it right. You need to be seen to respond to users concerns about spam, about it being hard to delete profiles. You need to post screenshots of the new nicer user interface you’re working on (here’s the new inbox! here’s a video of how the profile editor works).</p>
<p>Finally, I don’t know how the MySpace revenue model works, but they should invest in their advertising technology so that they can provide the best possible solution to their existing advertisers and attract new ones. The site needs to be profitable and strong on it’s own account and that should be by high quality, targetted and effective advertising that doesn’t detract from the user experience of the site.</p>
<p>So that’s my thoughts on building an a more agile, more profitable and more enjoyable Myspace. Competition is good and MS has the best opportunity of sticking it to FB.</p>
Thoughts from someone leaving NZ2011-01-24T00:00:00+00:00http://bnolan.github.com/2011/01/24/thoughts-from-someone-leaving<p>Since my girlfriend and I are leaving New Zealand for good probably, I decided I’d write about the New Zealand economy from my opinion as a 30 year old. Take this with a grain of salt, as it is purely anecdotal, I don’t have time to lookup citations for my observations.</p>
<h1 id="me">Me</h1>
<p>I’ve started two business in New Zealand. One, Indigo, generated about 100k€ of export revenue in the first year, employed 5 staff and is growing faster than ever. My co-founder has just left New Zealand with the rest of the company so they can be closer to their export markets in Europe. My other business, Zoomin, employed a half dozen people and was part of the Wellington startup cluster that jumpstarted in 2006. Rissa and I are now off travelling around New Zealand in a van, then moving to Brisbane for a few months before travelling to either the US or Europe.</p>
<h1 id="its-expensive-here">It’s expensive here</h1>
<p>Food and entertainment are expensive. It’s cheaper to shop at a Tengelmann supermarket in Munich using New Zealand dollars, than it is to shop at a New Zealand supermarket in New Zealand dollars. New Zealand cheese is cheaper in Australian supermarkets than in New Zealand supermarkets. It’s interesting talking to European backpackers who are astounded at how expensive it is to travel around in New Zealand. $42 / night for a campground? $8 for sausages? $5 for a loaf of bread? $18 for a six-pack of beer?</p>
<h1 id="housing-is-expensive-here">Housing is expensive here</h1>
<p>As an investment, I feel uncomfortable taking on an extra two years worth of salary as debt, when you don’t need to take that much debt to buy a house in another country. It seems especially perverse because there is so much good available land that is undeveloped. We’re not Japan or Germany.</p>
<h1 id="the-economy-is-overinvested-in-property-and-finance">The economy is overinvested in property and finance</h1>
<p>The UK learnt that relying purely on investment in property and the financial markets wasn’t a good idea over the past decade. They look at producing countries (like Germany, Japan and China) and realise that there is actual value in inventing technology, processes and processing businesses. New Zealand really only invests in the primary market and property. There are some well known exporters like Xero, but they’re about 0.01% of our industry. And New Zealand doesn’t seem to have realized yet that they need to become a strong economy in secondary industries (like processing, manufacturing, designing).</p>
<h1 id="new-zealand-should-have-a-distance-discount">New Zealand should have a distance discount</h1>
<p>Being so far from our customers, investors and partners, people should have a discount to work in New Zealand. Instead it’s more expensive to live in Wellington than to live in Munich.</p>
<h1 id="its-lovely-here">It’s lovely here</h1>
<p>I used to hate on New Zealand a bit much and I’m sorry for that. It’s really a beautiful country. The people are nice, the coffee is awesome. When things are slow, affordable and easy this is a wonderful country. The diving, beaches and mountain biking are off the hook.</p>
<h1 id="well-be-back">We’ll be back</h1>
<p>I hope that we’ll be back in New Zealand to live, or at least for our holidays. My prediction is that the property market will stagnate for half a decade, while salaries slowly creep up to bring affordability back. This relies on the government taking steps to squash investment property (since investment property tax advantages make an investment property a more affordable purchase than a first home).</p>
<h1 id="affordability-is-good-for-everyone">Affordability is good for everyone</h1>
<p>Part of the problem with the gap increasing between rich and poor, with people having to spend 120% of their salary to pay a mortgage, is that people are stressed out. I don’t want to live on a street full of renters who live in cold, substandard housing. I don’t want to live in a street of people who are working all hours and stressed out to meet their repayments.</p>
<h1 id="the-labour-government-was-atrocious">The labour government was atrocious</h1>
<p>I’m a Greens supporter, so in theory I should support Labour more than National, but Labour did absolutely nothing to protect New Zealands quality of life from the stupidity of the consumer debt bubble. It was John Key who did the first (half assed and limp wristed) attempts to redirect investment from the property sector into industry.</p>
<p>In a perfect (unlikely) world, property investment would be limited purely to apartment complexes and fresh developments in subdivisions, and tax-breaks would be geared to encourage owner-occupiers in existing properties.</p>
<p>Let’s see what happens.</p>
Heroku or Linode2011-01-20T00:00:00+00:00http://bnolan.github.com/2011/01/20/heroku-or-linode<p>The biggest problem I have with keeping my projects going is paying for and maintaining my servers. When you’re consulting, $60/mo for a dedicated server isn’t a big deal, but when it’s seeing no real traffic growth and you have to be responsible for backups, updates and admin - I find that after 3-4 months I usually mothball a project and have one less thing to administer.</p>
<p>Thus Twitterplaces, which was running on a dual-core 4gb box so it could index all the geo-tweets in the world. I left it run for about 6 months before taking it down. I’m planning on releasing weheartplaces to the appstore before I head off on holiday - so I need to set up a production environment for it. My options as far as I can tell are:</p>
<h2 id="heroku"><a href="http://heroku.com/">Heroku</a></h2>
<p><em>Pros:</em> Managed rails hosting, don’t have to do any admin, everything is backed up.</p>
<p><em>Cons:</em> Expensive (minimum of $50/month), no postgis.</p>
<h2 id="linode"><a href="http://linode.com/">Linode</a></h2>
<p><em>Pros:</em> Affordable ($20/month), can install whatever I like (Postgis / R / FastCGI scripts).</p>
<p><em>Cons:</em> Have to sysadmin and back it up myself.</p>
<p>I’m leaning towards linode at the moment since I already have one running diaspora-x.com, but I’d like some feedback on this. I know that at some level I should put my app on Heroku - but maybe I can have it running on my 512mb linode for a few months while I see if there is any uptake in weheartplaces at all?</p>
Quantitive measurement of bus drivers2011-01-19T00:00:00+00:00http://bnolan.github.com/2011/01/19/bus-drivers<p>I live in Wellington, New Zealand - and we’re unlikely to get modern tramways for a while, but we do have an extensive bus network. Busses are pretty average compared to trams, but I don’t think it has to be that way, and I’d like to propose some quantitive measurements that can be done to ensure the quality of drivers of the busses.</p>
<p>The main trick is to install a device in each bus that records the bus driver, the route, the timeliness of the route, comfort and fuel efficiency. You could use something like an android or iphone, since they have gps, accelerometers, a user interface and 3g connectivity.</p>
<h1 id="bus-driver">Bus driver</h1>
<p>The bus driver gets on the bus, types their name into the phone and they are logged in. They then specify which route they are running and the measurement system can work out where the bus should be.</p>
<h1 id="timeliness">Timeliness</h1>
<p>Measure the location of the bus against the measured stops. Use this data to show a timeliness report (as they do in Melbourne) and use this data to decide what changes need to be made to routes, traffic lights and bus lanes. Instead of drivers having to rely on anecdotal evidence, they can show why changes need to be made to each route, for example, a sector may need to be made longer or shorter, route timings changed.</p>
<p>This could be done on an ongoing basis as traffic patterns change season by season.</p>
<h1 id="comfort">Comfort</h1>
<p>This is a big one. The major problem with busses after timeliness, are bad drivers. The main factors of a bad driver:</p>
<ul>
<li>
<p>Accelerate really hard, causing a huge amount of noise. If you’ve ever seen a baby sitting in a pram on lambton quay when one of the old diesel busses accelerates away you’ll know what I mean. This is uncomfortable for passengers, adds unneeded noise to the city, is dangerous for people on the street, promotes aggressive driving, and wastes fuel.</p>
</li>
<li>
<p>Drive too fast through corners. Some bus drivers drive their busses like race cars. And some driver their busses really carefully and considerately. The considerate drivers need to be rewarded.</p>
</li>
<li>
<p>Break too hard into a corner, don’t predict the lights and coast to a stop.</p>
</li>
</ul>
<p>By measuring the gps location and instantaneous acceleration 10 times a second, with the phone, or measurement device, mounted in a rigid cradle at the front of the bus, it would be possible to create a measurement for how smooth and efficient a bus driver was.</p>
<p>It would be easy to measure when a bus driver wasn’t driving smoothly and comfortable.</p>
<h1 id="learning-and-improving">Learning and improving</h1>
<p>I’m not sure what the best way to use this data would be. It seems like paying bonuses or penalizing bad drivers would be the most effective way to ensure a competition for smoothest driver, but I imagine there are kinder ways of rewarding correct behaviour.</p>
<p>It would be very important for all measurements to be compared correctly against other drivers on the same route, on the same bus, and at the same time of day, as some routes would be by necessity more intense than others. If you’ve taken the bus through vogeltown or roseneath you’ll know it’s a pretty windy path.</p>
<p>It would also be important for the data to be used to make changes at the roading and management level as well, instead of just passing the buck onto bus drivers and expecting them to do miracles. Wellington City Council should improve busways where needed, relying on real data, not anecdotes. And the bus company management should be made accountable as well.</p>
<p>Especially for buying the bad old MAN busses that are wheelchair-hostile and noisy as hell.</p>
<h1 id="public-data">Public data</h1>
<p>Finally, all this data should be publically available (in an suitably anonymized, aggregate form) so that ratepayers know that they are getting value for money from their subsidised bus service.</p>
<p>(If anyone actually wants this thing to be built and exist, I’d recommend getting a local company like youdo, southgate or silverstripe to build it)</p>
Weheartiphone2011-01-18T00:00:00+00:00http://bnolan.github.com/2011/01/18/weheartiphone<p>I got inspired by the design of <a href="http://itunes.apple.com/nz/app/plaintext-dropbox-text-editing/id391254385?mt=8">plaintext</a> and redesigned my phonegap app. Weheartplaces is an app for bookmarking places and accessing them on the go. It syncs with your facebook account so that in the future you’ll be able to see your friends bookmarks and send bookmarks to friends.</p>
<p><img src="/images/weheart.png" /></p>
Koala for facebook2011-01-17T00:00:00+00:00http://bnolan.github.com/2011/01/17/koala-for-facebook<p>I’ve been using <a href="https://github.com/arsduo/koala/wiki/">Koala</a> + Facebook connect for facebook development in rails. It’s an awesome tool for exploring the graph API using pure rails. You don’t need to use devise to do your authentication, just facebook connect, then take the access token and then you have access to the facebook api as follows:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>@oauth = Koala::Facebook::OAuth.new(app_id, code, callback_url)
@oauth.get_user_info_from_cookies(cookies)
</code></pre>
</div>
<p>It’s interesting when deciding where to use APIs like facebook or the google maps geocoder api. Should you do it client side, and reduce load on your server, but add the complexity of handling state and large amounts of data in an environment that’s not really designed for it - or do you do it client side and increase your ops cost?</p>
<p>It’s still a hairy one I haven’t decided. One of the classic problems is when people search on a social mapping site (like weheartplaces):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/places/search?query=Eiffel+Tower
/places/search?query=Nelson,+New+Zealand
</code></pre>
</div>
<p>You want to look for any places called <code class="highlighter-rouge">eiffel tower</code> in your database (probably using tsearch2), but you also want to run the search through a geocoder to see if the user actually searched for a city. And you need to do some kind of combinatorial weighting function to work out what was the most likely result for a given query.</p>
<p>The best way to do that is to gather data on the server side so that you can train your weighting function based on previous results and what the user was actually looking for.</p>
<p>Pure javascript and focus on the UI? Or a strong back-end service and focus on the data quality? Or both? And never finish the project.</p>
<p>;D</p>
Capt in production2011-01-14T00:00:00+00:00http://bnolan.github.com/2011/01/14/capt-production<p>I’ve added some features to <a href="http://github.com/bnolan/capt">capt</a>, it’s still very rough and needs a lot more work, but it’s coming together. Added recently:</p>
<h1 id="build-command">Build command</h1>
<p>Run <code class="highlighter-rouge">capt build iphone</code> to create a build in <code class="highlighter-rouge">builds/iphone</code> that will compile all coffeescript, concatenate and minify all scripts, compile all the less into css and run the yui compressor over it, then create an index.html from your index.jst, with only two files as an include.</p>
<h1 id="added-support-for-targets-in-the-configyml">Added support for targets in the config.yml</h1>
<p>I have this on my config.yml:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>iphone:
lib/phonegap.js
platforms/iphone.coffee
</code></pre>
</div>
<p>These are included only when developing against the iphone, and are all merged into one file for production. I do the same for nokia (which requires json.js and some special code to deal with the nokia webruntime).</p>
<h1 id="more-test-support">More test support</h1>
<p>I’ve improved the testing templates, but I still need to get a command line test environment running - basically type <code class="highlighter-rouge">capt test</code> and have it run the whole test suite using node.js and jsdom.</p>
<h1 id="future-of-capt">Future of capt</h1>
<p>I doubt the first decent release of capt will be prior to April, since I’m heading away on a campervan holiday of new zealand for 2 months, but keep an eye out then, and feel free to fork and send pull requests in the meantime.</p>
Jquery Mobile or not2011-01-13T00:00:00+00:00http://bnolan.github.com/2011/01/13/jquery-mobile-or-not<p>I’ve been working on <a href="http://weheartplaces.com/">weheartplaces</a> again, trying to get it into the appstores and out the door. I used capt to build the bundled and compressed javascript, and using jquery mobile, the closure compiled code was 161k.</p>
<p>I removed jquery and jquery mobile and replaced it with zepto.js. It took about an hour or so to recode a minimal CSS theme that replaces all the jquery mobile functionality (with the exception of fixed position viewports) and the resulting closure compiled code was 41k.</p>
<p>The huge difference was in application startup time, and in less crashes on the actual device. On the simulator, the javascript compiles instantly, but on my 2g iphone (same as a 3g iphone processor), the app starts in 4 seconds, instead of the previous 8. It’s also smoother page to page and the user interactivity is quicker.</p>
<p>jQuery mobile is fantastic and I may end up going back to it - but for now I’m going to do my mobile development using jquery.js (zepto doesn’t work on Nokia), backbone.js and a custom stylesheet. I’ll put up some screenshots of the new weheartplaces mobile app shortly.</p>
2011 - the year of pure javascript apps2011-01-12T00:00:00+00:00http://bnolan.github.com/2011/01/12/pure-javascript-apps<p>Pure javascript apps, or apps that use javascript for all the presentation have been coming to prominence for years.</p>
<ul>
<li>Google maps 2007</li>
<li>Phonegap 2008</li>
<li>Facebook 2009</li>
<li>Twitter 2010</li>
</ul>
<p>It’s an obvious prediction, but 2011 will probably have a lot more javascript apps being released. From mobile - where phonegap is so important to smaller companies that can’t afford to redevelop an app natively in Java and Objective C - through to ‘desktop’ web apps like the twitter client.</p>
<p>Some of the key technologies that I peg for uptake this year:</p>
<h1 id="coffeescript">CoffeeScript</h1>
<p>I’m a big fan of this. It’s easy to learn (if you’re open minded and don’t try to be too clever), and makes javascript immeasurably more pleasurable to write. It’s basically a whitespace aware javascript compiler, so it does away with all the curly brackets, replaces <code class="highlighter-rouge">function</code> with <code class="highlighter-rouge">-></code> and has very nice built in prototype-based class support.</p>
<h1 id="couchdb">CouchDB</h1>
<p>This one I’m not so sure about. Couch is really amazing for creating pure javascript apps, since you don’t require a middleware, you can put business logic into the database layer (rules are written in javascript) and they talk JSON to your javascript app. The downsides with couch are that it is a bit of a mindfuck to get used to, coming from a relational background. Also, the couchapps / authentication / client-facing parts of couch are relatively new and I think there needs to be a lot of work on documentation and frameworks to make client-facing couch apps easier. So, couch for indie developers making pure javascript apps - watch out for this.</p>
<h1 id="backbonejs">Backbone.js</h1>
<p>There are a dozen or so javascript MVC frameworks. I imagine they’ll all take off (javascriptmvc, backbone, ext..), but the one I really like is backbone. It’s pure javascript, plays nicely with Zepto (a tiny implementation of jquery for webkit-mobile) and works well with coffeescript. It’s very event based, so some parts are really great (like redraweing the UI when an item in a collection changes), but it’ll be interesting to see how it progresses over the year, hopefully some great tutorials come out of it and it’ll become a dominant force in javascript development.</p>
<h1 id="phonegap">Phonegap</h1>
<p>I know that mobile apps written in phonegap are seldom as good as native apps, and a well written native app will always be better than an html+css app, but phonegap is just going to get better and better this year. If you look through the <a href="https://github.com/purplecabbage/phonegap-plugins">phonegap plugins</a> for each platform, there are some great bits, like the email connector and child browsers.</p>
<h1 id="nodejs">Node.js</h1>
<p>To finish off, node.js. Node is a bit of a mixed bad. It’s awesome because everything is async, it compiles easily, is very fast and seems to be reliable. On the downside, the API changes like a bitch, constantly. Any code you wrote for node 6 weeks ago, probably won’t work anymore. I appreciate that it’s a pre 1.0 project, and they’re trying to work out how to do everything in the best possible way, but if development really needs to stabilise around some core areas soon (like socket programming).</p>
CouchDB OAuth2011-01-11T00:00:00+00:00http://bnolan.github.com/2011/01/11/couchdb-oath<p>I think couchdb is a good solution for creating pure javascript apps, but there are a few big problems with using a service like clouddb, particulary around authentication. There already exists a couchdb oauth module, but afaict, it’s for authenticating against couchdb, not for creating a couchdb account against a twitter or facebook account.</p>
<p>If someone like couchone added a simple way to implement an oauth consumer in their apps, it’d be a really fantastic way to prototype apps. You could authenticate using javascript against twitter, google or facebook - create the users account and database, and then return them to your application, all without writing any server side code.</p>
<p>The current couchdb authentication is a bit basic and doesn’t support password reset or email verification, and you can’t really build such things as couch apps.</p>
Couchdb behind Apache2011-01-10T00:00:00+00:00http://bnolan.github.com/2011/01/10/couchdb-behind-apache<p>I had some issues getting couchdb to work correctly behind Apache. The big problem I was getting was that cross-domain requests were being “preflighted” where the browser sends an <code class="highlighter-rouge">OPTIONS</code> request to the server, which was being proxied to Couchdb - which was returning error 405, since couch doesn’t support the OPTIONS method.</p>
<p>Here is the complete virtualhost setup for giving your localhost apps cross-domain access to couchone.</p>
<div class="highlighter-rouge"><pre class="highlight"><code><VirtualHost *:80>
ServerName mycouch.com
Header set Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Headers Content-Type
Header set Access-Control-Allow-Methods "GET, PUT, OPTIONS, DELETE, POST"
Header set Access-Control-Max-Age 3600
ProxyPass / http://mycouch.couchone.com/ nocanon
ProxyPassReverse / http://mycouch.couchone.com/
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^OPTIONS
RewriteRule .* /index.html [L]
</VirtualHost>
</code></pre>
</div>
Backbone and couchdb2011-01-07T00:00:00+00:00http://bnolan.github.com/2011/01/07/backbone-couchdb<p>I’ve been using the <a href="https://github.com/janmonschke/backbone-couchdb/">backbone couchdb</a> connector by Jan and I’ve been impressed. The real showstopper is realtime updates using the changes feed from couchdb.</p>
<h1 id="background">Background</h1>
<p>For complex geospatial apps, you can’t go past rails + postgis as a toolchain, but for simpler apps where you do all of the work on the front end and basically need a hash store, rails is overkill.</p>
<p>Instead of having to run <code class="highlighter-rouge">rake migrate</code> everytime you want to add a column, you just write to the backbone model, call save and it’s serialized away into your document store.</p>
<h1 id="schemaless-stores">Schemaless stores</h1>
<p>It’s possible to use postgres in a schemaless way by either:</p>
<ul>
<li>Autogenerating columns as they are referred to</li>
<li>Serializing all the columns into a blob</li>
</ul>
<p>But it makes sense to investigate databases that are designed for the purpose. The two I have experience with are:</p>
<ul>
<li>MongoDB</li>
<li>CouchDB</li>
</ul>
<p>Both are good solutions, I’ve used MongoDB extensively with node and it’s json support is very good. I was interested in trying out couch since it is an almost pure-javascript solution, with javascript views and support for couchapps.</p>
<p>CouchDB also runs well without any middleware, your app talks directly to the database.</p>
<h1 id="backbone-couchdb">Backbone-couchdb</h1>
<p><a href="http://documentcloud.github.com/backbone/">Backbone</a>, if you haven’t seen it - is an excellent tool to give structure to your javascript applications. I use it extensively. One of the great benefits of backbone is the use of models and collections, both of which emit events when their contents change.</p>
<p>So say you have a list of <code class="highlighter-rouge">posts</code> on the left side of your page, you bind your <code class="highlighter-rouge">render()</code> (or redraw) function to the <code class="highlighter-rouge">add</code> and <code class="highlighter-rouge">remove</code> events of the Posts collection and the list will always be up-to-date. For example - if you opened your javascript console and typed:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Posts.add(new Post(content : "Hello world!"))
</code></pre>
</div>
<p>All the dom elements will be automatically created and the posts list redrawn.</p>
<p>Backbone by default doesn’t come with any persistence layer. Models and collections are stored as javascript variables and they are lost when the page is refreshed. You can persist using the build-in sync methods that submits json to a server. Backbone uses the rails idioms for this, although it’s easy enough to update Backbone.sync to work with other datastores / application frameworks.</p>
<p>You can also use the localStorage plugin to persist your models to the browser. The <a href="https://github.com/janmonschke/backbone-couchdb">backbone-couchdb</a> persists your models to the couchdb store, so whenever you call save, it posts your model to the server - and when you reload the page, it loads all your models from the server.</p>
<p>And because it’s couchdb, all you have to do is create the database record in futon (one click) and all the models are automatically saved away, the database doesn’t need to be aware of what you’re saving, it just saves.</p>
<h1 id="the-magic-of-realtime">The magic of realtime</h1>
<p>The magic part of this is the <code class="highlighter-rouge">_changes</code> feed from couchdb. A client can request the _changes feed by doing an ajax request. Couchdb will then block (ala long polling) until there are changes on the database, at which point it will send down json and close the connection. The client processes these changes (updating collections and models as needed) and then reopens the connection to the changes feed.</p>
<p>And because you are already re-rendering your UI whenever a collection changes, you can see in realtime what anyone else is doing on the same dataset that you are using.</p>
<h1 id="couchone">Couchone</h1>
<p>I’m using <a href="http://couchone.com/">couchone</a> to host my couch databases. It’s a great service, the only downside is that you can’t set <a href="http://metajack.im/2010/01/19/crossdomain-ajax-for-xmpp-http-binding-made-easy/">Access-Control-Allow-Origin</a> headers, which would let you do cross-domain ajax calls (so you can host your app at github and store your database at couchone). I got around this by proxing through my linode box, but this kills performance. Hopefully the couchone guys will give you the option to change the headers (they could use varnish or similar).</p>
Diaspora-X speedups2010-12-30T00:00:00+00:00http://bnolan.github.com/2010/12/30/diaspora-x-speedups<p>I’m back from a week on the Australian beaches (I recommend surf lessons at Coolum and the breaks down at Burleigh heads) and at a wicked little festival (Blah blah blah is a highly recommended party), so it’s time to fix up a few issues that were raised while I was away.</p>
<p>About two weeks ago I wrote <a href="https://github.com/bnolan/diaspora-x2">Diaspora x²</a>, a port of the Diaspora UI to pure Javascript. My original Diaspora-xmpp port used rails and xmpp4r to federate by XMPP. It was a good idea, but I’m much more at home using javascript that rails, and there was a lot of work required to get the xmpp connector working reliably at high loads.</p>
<p>That’s why I got strophe.js, which is a javascript library for doing xmpp over the http transport, <a href="http://xmpp.org/extensions/xep-0206.html">bosh</a>. The result was diaspora-x2, which is now online at diaspora-x.com.</p>
<p>All well and good - but the first release was REALLY slow. Todays fixups:</p>
<h2 id="queue-rendering-calls">Queue rendering calls</h2>
<p>My views watch the posts collection for add and remove events. So an xmpp message comes in, it gets added to the posts collection, and then the view re-renders. The problem is, the way bosh batches messages together means that I often iterate over a 100+ messages, and add them one by one to the collection. This then triggers a 100+ <code class="highlighter-rouge">render()</code> calls, each of which constructs a big string, the string is parsed, then the elements are added to the dom, then they’re all deleted and it starts again.</p>
<p>Super easy fix that it applicable to a lot of backbone apps:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>render: ->
@el.html(@template())
</code></pre>
</div>
<p>Replace with:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>render: ->
if @renderTimeout
clearTimeout @renderTimeout
@renderTimeout = setTimeout( =>
@el.html(@template())
, 50)
</code></pre>
</div>
<p>The only downside to this is that you have to make your view tests async.</p>
<h2 id="better-rendering">Better rendering</h2>
<p>The bigger problem is that because the entire posts wall gets re-rendered everytime theres an update means that anything you were typing into an inline textbox will get annihilated. I’ll be fixing this by progressively inserting and removing dom elements when the collection changes. Ordering will be done by having a <code class="highlighter-rouge">data-created-at</code> attribute on the elements, so you select the dom elements, then move down the list until you get to the correct element and insert.</p>
<h2 id="embedly">Embed.ly</h2>
<p>This will let me re-enable embed.ly, so that we can have cool inline media, like soundcloud tracks, youtube videos or flickr photos.</p>
<h2 id="media-upload">Media upload</h2>
<p>I want to enable <a href="http://html5doctor.com/native-drag-and-drop/">drag and drop</a> media sharing, where the user selects which service they want to host the media. By default we could support imgur, then the user drag/drops images up and they get uploaded to imgur and then post the text link.</p>
A good cofounder2010-12-19T00:00:00+00:00http://bnolan.github.com/2010/12/19/a-good-cofounder<p>I’ve had the good fortune to have had a co-founder in both of the startups I’ve been involved with. Given the option, I would always elect to start a business with someone else.</p>
<p>However - before you meet your next cofounder, there will be a period of time where you work on your own idea. Before I founded Zoomin (which we almost, but not quite, sold to a multinational), I worked on the prototype (a social mapping site) for about 4 months.</p>
<p>In a lot of projects, you will have to be a one man band for a while. You’ll have to promote yourself, develop the software, motivate yourself everyone morning.</p>
<p>From prior experience, I know that I can do all of these:</p>
<ul>
<li>Talk to people and investigate the market (research)</li>
<li>Code, architect and design (development)</li>
<li>Sysadmin, keep records and pay taxes (operations)</li>
<li>“Do marketing” and write blog posts (promotion)</li>
<li>Set a price and ensure people pay (revenue)</li>
</ul>
<p>The things is… <em>I don’t think anyone who tries to build a business ever realizes how hard it is going to be - before they set out.</em></p>
This week2010-12-17T00:00:00+00:00http://bnolan.github.com/2010/12/17/this-week<p>It’s time to drink a beer and write a blog post. This week I worked on diaspora-x squared, an xmpp client written using javascript. It uses strophe.js and backbone.js. I had a few issues that needed attention with backbone:</p>
<h1 id="multiple-render-calls">Multiple render calls</h1>
<p>I was receiving xmpp stanzas and inserting them into a backbone collection. The collection was firing the <code class="highlighter-rouge">add</code> event and I had hooked that to call <code class="highlighter-rouge">this.render</code>. That meant that I was calling render (and the browser was parsing html, creating dom nodes and reflowing the layout) about 30 times in a row, which is fucking inefficient (even though the browser never rendered the nodes).</p>
<p>This could be fixed by putting in a 20ms timeout before calling <code class="highlighter-rouge">this.render</code>, and canceling the timeout when a subsequent update request came in (you need to make the timeouts at least 20ms or some browsers will call the timeout immediately).</p>
<h1 id="localstorage">LocalStorage</h1>
<p>I didn’t implement localStorage to cache the previously received xmpp stanzas, mainly because there is no localStorage support built into backbone. I know that backbone is meant to stay lightweight and tiny, but it would be nice if the localStorage store was “officially supported”, then I’d use it more often.</p>
<h1 id="confusion-about-submitting-forms">Confusion about submitting forms</h1>
<p>When I write form submission code, I do it in a view - so the view has a <code class="highlighter-rouge">submit</code> method which is hooked using the delegateEvents method on form submission. I end up processing the form entry, saving the model, adding it to a collection and then doing a redirect (using window.location.hash), all in the view. Some of this code feels like it should be in the controller, but I’m not sure how to do that. Maybe I’m just getting confused with the rails way of doing things.</p>
<h1 id="adding-production-support-to-capt">Adding production support to capt</h1>
<p>I’ve used <code class="highlighter-rouge">capt</code> (my code generation / development mode server tool for creating backbone apps) to build two apps now and it really is a good tool. I need to add:</p>
<ul>
<li>Production mode (concatenate files in order, run through closure)</li>
<li>Server side coffeescript processing (not really needed, but a nice to have, esp. with growl support)</li>
<li>Working test framework (I am generating test cases but haven’t been running them, need an easier way to do so)</li>
<li>Documentation and promotion (so that people learn about it)</li>
</ul>
<p>Interesting week all the same. I had two really good days in the flow (2000 loc on thursday, about 800 loc today). Invented two nice things, an xmpp-pubsub social network (diaspora-x2) and a bookmarklet that scrapes geodata from a currently open page (for adding places to weheartplaces).</p>
<h1 id="two-levels-deep">Two levels deep</h1>
<p>In my weheartplaces bookmarklet, I use this bit of code to search the dom for any google maps instances in the first two layers of the dom, and then extract the lat/long of the map and use it for the bookmarklet:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if google? and google.maps? and google.maps.Map
maps = []
# Search two levels deep...
for k,v of window when v instanceof Object
if v instanceof google.maps.Map
maps.push v
else
for i,j of v when j instanceof Object
if j instanceof google.maps.Map
maps.push j
for map in maps when map.getCenter
try
attributes.geo.push [map.getCenter().lat(), map.getCenter().lng()].join(", ")
catch e
# ...
for map in maps when map.getCenterLatLng
try
attributes.geo.push [map.getCenterLatLng().lat(), map.getCenterLatLng().lng()].join(", ")
catch e
# ...
</code></pre>
</div>
<p>Works really well. Quite stoked. I’ll try and release the weheartplaces bookmarklet next week.</p>
Attacked by wheelchair!2010-12-14T00:00:00+00:00http://bnolan.github.com/2010/12/14/attacked-by-wheelchair<p>Bwahahaha <a href="http://bit.ly/f9SzAH">the bbc</a>. The world media is really doing a great job of acting like fucks at the moment. My favourite quotes from this video: “There are allegations you were rolling your wheelchair towards the police” and (to the cerebal palsy sufferer) “There were people throwing chunks of concrete at the police. Were you throwing things at the police”.</p>
<object width="360" height="240"><param name="movie" value="http://www.youtube.com/v/tXNJ3MZ-AUo?fs=1&hl=en_US" /></param><param name="allowFullScreen" value="true" /></param><param name="allowscriptaccess" value="always" /></param><embed src="http://www.youtube.com/v/tXNJ3MZ-AUo?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="240" /></embed></object>
<p>Those riot police do earn their money though. Just imagine standing there, in the open, having a guy in a wheelchair rolling towards you.</p>
<p>Fucking terrifying.</p>
<p>To be honest, the police did have riot gear to protect themselves.</p>
<p>And there were hundreds of them.</p>
<p>But the dude had a FUCKING WHEELCHAIR!</p>
<p>Those wheelchairs are pretty much a deadly weapon.</p>
Capt, a tool for backbone.js2010-12-13T00:00:00+00:00http://bnolan.github.com/2010/12/13/capt-a-tool-for-backbone<p>I’ve been working on <a href="http://github.com/bnolan/capt">capt</a> for a few days now, so I wanted to write an introductory post. Capt is a tool for generating and serving backbone.js projects in development mode. It’s currently tooled for my personal toolchain:</p>
<ul>
<li>jQuery</li>
<li>Backbone.js</li>
<li>Underscore.js</li>
<li>QUnit</li>
</ul>
<p>You can imagine capt as a combination of jamit and the rails code generators. For example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> $ capt new blogproject
* Creating folders
* Downloading libraries
</code></pre>
</div>
<p><cite>Create a new project</cite></p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ find blogproject
blogproject
blogproject/app
blogproject/app/controllers
blogproject/app/models
blogproject/app/views
blogproject/app/views/jst
blogproject/config.yml
blogproject/index.jst
blogproject/lib
blogproject/lib/backbone.js
blogproject/lib/coffeescript.js
blogproject/lib/jquery.js
blogproject/lib/underscore.js
blogproject/public
blogproject/public/stylesheets
blogproject/test
blogproject/test/controllers
blogproject/test/fixtures
blogproject/test/models
blogproject/test/qunit.css
blogproject/test/qunit.js
blogproject/test/views
</code></pre>
</div>
<p><cite>File structure</cite></p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ capt generate model post
Created app/models/post.coffee
Created test/models/post.coffee
Created test/fixtures/post.yml
$ capt generate model comment
Created app/models/comment.coffee
Created test/models/comment.coffee
Created test/fixtures/comment.yml
</code></pre>
</div>
<p><cite>Create models and tests</cite></p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ capt server
node-router server instance at http://*:3000/
</code></pre>
</div>
<p><cite>Start a webserver in production mode</cite></p>
<p>Capt is written in coffeescript and uses node.js (I’ll distribute it by npm once it’s done), but at the moment is just for client work, it does server side processing to automatically include the correct files, but you’re not meant to develop node apps with it, it’s for developeing client apps.</p>
<p>You can then browse to /index.html and it’ll automatically include all the correct files in the order specified in config.yml. You can also browse to /test/ and it’ll run the auto generated tests.</p>
<p><img src="/images/capt-test.png" /></p>
<p><cite>The default generated tests running in qunit</cite></p>
<p>It’ll be a while until there’s a good first release (I might take the current random code and covert it to use Cake for example), but here’s a headsup if anyone it working on something similar.</p>
<h1 id="production-targets-and-textmate">Production targets and textmate</h1>
<p>Once I’ve got things going - I want to have a few production targets:</p>
<ul>
<li>Nokia WRT</li>
<li>Phonegap</li>
<li>HTML5 standalone w/ manifest</li>
<li>Web deployment</li>
</ul>
<p>With a one line command to package up the project into one closure-ified project. I also want to create a textmate bundle for running the tests from within textmate by hitting command+r - like you can do with rspec. I’d like the tests to run within the node jsdom project, so you don’t need to remote control a browser, but I’m not sure how well that will work on large projects.</p>
<p>Anyway - I’ll keep you updated.</p>
Visa is down2010-12-09T00:00:00+00:00http://bnolan.github.com/2010/12/09/visa-is-down<p>Visa.com and mastercard.com are down (as of 3pm Thursday in New Zealand) from the <a href="https://github.com/NewEraCracker/LOIC/downloads">anonymous</a> <a href="http://en.wikipedia.org/wiki/Denial-of-service_attack">ddos</a>. For my part, I made a protest sign that shows my feeling about how Sweden is acting as an independent and sovereign nation with respect to Julian Assange:</p>
<p><img src="/images/sweden.png" />
<cite>Sweden, Americas Bitch</cite></p>
Social network over XMPP2010-12-08T00:00:00+00:00http://bnolan.github.com/2010/12/08/Social-network-over-xmpp<p>I thought I’d do a post showing how a social network over XMPP works. This isn’t any invention of mine, I’ll just point to the relevant specs and give my understanding of how it can work. I’m not an XMPP <a href="http://www.buddycloud.com/">expert</a> though, so take this with a pinch of salt.</p>
<h1 id="user-ids">User IDS</h1>
<p>Users are identified by their jabber id, which looks like an email address - eg: <code class="highlighter-rouge">bnolan@gmail.com</code> or <code class="highlighter-rouge">ben@diaspora-x.com</code>. Users have to be associated with a jabber server, but they can use any client they want. For example I could use diaspora-x.com to access my bnolan@gmail.com jabber account.</p>
<p><em>nb:</em> Diaspora-x won’t support gmail as a jabber server until gtalk supports <a href="http://www.google.com/support/forum/p/Talk/thread?tid=2c8f523795126684&hl=en">oauth</a>, since otherwise a diaspora-x seed would have to store google account passwords.</p>
<h1 id="pubsub">Pubsub</h1>
<p>XMPP supports publish/subscribe via <a href="http://xmpp.org/extensions/xep-0060.html">XEP-0060</a>. This means that you can send a message to your xmpp server, with no to address specified. Everyone on your roster who is subscribing for those kind of events will get your message. If someone on your roster is offline, their xmpp server will store your message until they come back online.</p>
<h1 id="personal-eventing-protocol">Personal eventing protocol</h1>
<p><a href="http://xmpp.org/extensions/xep-0163.html">XEP-0163</a> describes how any jabber user can act as a virtual pubsub service, and send their own status / events.</p>
<div class="highlighter-rouge"><pre class="highlight"><code><iq from='juliet@capulet.lit/balcony' type='set' id='pub1'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<publish node='http://jabber.org/protocol/tune'>
<item>
<tune xmlns='http://jabber.org/protocol/tune'>
<artist>Gerald Finzi</artist>
<length>255</length>
<source>Music for "Love's Labors Lost" (Suite for small orchestra)</source>
<title>Introduction (Allegro vigoroso)</title>
<track>1</track>
</tune>
</item>
</publish>
</pubsub>
</iq>
</code></pre>
</div>
<p><cite>Sending a song update using the tunes extension</cite>
<br /></p>
<p>This pubsub activity is commonly used by <a href="http://xmpp.org/extensions/xep-0118.html">user tunes</a> so you can see what your friends are listening to. If you’ve ever seen someone updating their adium status with song names from itunes, they might’ve been using XEP-0118.</p>
<p>Another use of publish / subscribe is microblogging - described in <a href="http://xmpp.org/extensions/xep-0277.html">XEP-0277</a>.</p>
<h1 id="microblogging">Microblogging</h1>
<p>Microblogging over xmpp create a node <code class="highlighter-rouge">urn:xmpp:microblog:0</code> on each user, and then provides a protocol for sending atom messages over xmpp. This protocol has been expanded upon by the <a href="http://http://onesocialweb.org/">Vodafone research group</a> as <a href="http://onesocialweb.org/spec/1.0/osw-activities.html">Activity Streams over XMPP</a> which lets you publish, update and delete posts. It also gives a <a href="http://activitystrea.ms/schema/1.0/activity-schema-01.html">vocabulary of verbs</a> that you can use to describe your activity, so you can like, bookmark or group posts.</p>
<p>A client doesn’t have to implement all the verbs. Diaspora*x only implements the like and post protocols (replies to posts are posts as well), but as you fill out the functionality, you have a predefined vocabulary so that clients can work together.</p>
<h1 id="alternate-clients">Alternate clients</h1>
<p>I have a few goals for diaspora*x, if I keep working on it.</p>
<ol>
<li>Get federation working reliably with other servers, currently the xmpp connector is incomplete, although it shouldn’t take long to get going. I’d also like to test the federation against another social network, maybe buddycloud, or onesocialweb.</li>
<li>Create a desktop client using strophe.js, that connects to the jabber server directly (via BOSH) and shows how a client can be implemented in the simplest possible way.</li>
</ol>
<p>A part of my job I think, is to demistify social networking over xmpp, as it’s quite a simple idea and easy to implement on the current specs.</p>
<p>I also have to read up on ostatus and work out if and how diaspora*x should work with ostatus. If it makes sense to add ostatus support, that would mean anyone on the status.net network would be friendable via a disapora*x seed.</p>
Diaspora*x2010-12-07T00:00:00+00:00http://bnolan.github.com/2010/12/07/diaspora-x<p>I like Facebook, and at the moment I think it does a great job of managing privacy on the Facebook site. My only concern is that of censorship. If the US government asks Facebook to censor something, Facebook has to acquiesce to their request.</p>
<p>I don’t believe this is a problem in the short term (if the wikileaks facebook page goes offline no one is going to die), but I do think the idea of having a distributed social network - where there is no central point of control or central point of failure - is a cool idea.</p>
<p>So I was excited to learn about Diaspora. It’s clearly something people are interested in, their fundraising was 20x oversubscribed. So when they did a public release, I was keen to check out the code and try it out.</p>
<p>The current alpha release of Diaspora is interesting, but it clearly has a ways to go before it’s ready for day-to-day use. The biggest problem I found with it is that it doesn’t support XMPP for communication between different seeds. XMPP is also known as Jabber, and it’s the protocol that gtalk and ichat use to communicate. Even facebook uses <a href="http://developers.facebook.com/blog/post/110">xmpp</a> in places to allow you to interact with facebook chat.</p>
<h1 id="adding-xmpp-support-to-diaspora">Adding XMPP support to Diaspora</h1>
<p>So I grabbed a few bits out of the Diaspora trunk and built a <a href="http://github.com/bnolan/diaspora-x">little app</a> to demonstrate how a basic distributed social network could work. You can try out my result of a few days coding here:</p>
<h2 id="-diaspora-xcom">» <a href="http://diaspora-x.com/">diaspora-x.com</a></h2>
<p>I’m not sure how much more work I’m going to do on this project, I’ve proven the idea works to myself, and I’d like Diaspora proper to start work on XMPP support for the main project.</p>
<p>However - if you want me to add a feature to the diaspora-x website, put your requests into the <a href="https://github.com/bnolan/diaspora-x/issues">github issue tracker</a> and I’ll see what I can do.</p>
<p>There are installation instructions <a href="https://github.com/bnolan/diaspora-x/blob/master/doc/install.md">here</a> if you want to try running Diaspora*x yourself.</p>
Offending people2010-12-06T00:00:00+00:00http://bnolan.github.com/2010/12/06/offending-people<p>Any time you click a broken link (like this one for the <a href="https://joindiaspora.com/2010/10/29/october-update.html">october update</a>) on the <a href="http://www.joindiaspora.com/">Diaspora</a> site, you get this guy staring back at you.</p>
<p><img src="/images/diaspora-404.png" />
<cite>Oops. My bad.</cite></p>
<p>Except, well I didn’t do anything wrong, except I clicked a link that you broke on your site. And now I feel like a fool.</p>
<p>Cool graphic design. Really bad user experience.</p>
On how things stay the same2010-11-30T00:00:00+00:00http://bnolan.github.com/2010/11/30/things-stay-the-same<p>When you’re young and you have problems or obstacles, you are comforted by the fact that you’ll get through this and in the future you won’t have such problems.</p>
<p>It’s interesting how no water how as you get older and more experienced, you get much better at solving your problems (oh it’s a X so i’ll do a Y), but that feeling of ‘my god, what the fuck am i going to do’ doesn’t go away. It just is more brief before you accept ‘oh well, i’ll just have to use a self tapping bit and use an oversized thread.</p>
<p>Well, that’s how it’s been so far - be interesting to see what the next 20 years of problem solving will reveal.</p>
Horizontal sharding on Postgres2010-11-29T00:00:00+00:00http://bnolan.github.com/2010/11/29/horizontal-sharding-postgres<p>Working on <a href="http://twitterplaces.com/">twitterplaces</a> I get around 2 million tweets per day. The importer is a streaming curl session that I pipe into a ruby script that analyzes the tweet and sticks it into a postgres database.</p>
<h1 id="tweet-analysis">Tweet analysis</h1>
<p>Some of the analysis I do for each tweet:</p>
<ul>
<li>Is it an @ reply?</li>
<li>What place_id is being referred to?</li>
<li>Does the user seem spammy?</li>
<li>What neighbourhood does this tweet belong to?</li>
<li>What city does this tweet belong to?</li>
</ul>
<p>This has worked really well and even though its low tech, when combined with a monit script, it’s easy to control and robust.</p>
<h1 id="the-problem">The problem</h1>
<p>The problem is that after 20 days, you have more than 50 million tweets and postgres starts to grind to a halt when inserting data.</p>
<p>If you had forgotten all of your university level mathematics - you might model the relationship between time to insert and the number of tweets like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lim T(insert_tweet) as SUM(tweets) &rarr; 50m = &#8734;
</code></pre>
</div>
<h1 id="the-solution">The solution</h1>
<p>The obvious solution is to delete the old tweets out of the database. Maybe a cron job like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>delete from tweets where created_at > 7.days.ago
</code></pre>
</div>
<p>Except every time you do that (and the VACUUM that you have to run afterwards), postgres will stop responding for about an hour, giving a theoretical uptime limit of 0.95. Since I’m aiming for a uptime of at least 3 sixes (0.9666), I decided to look int other options.</p>
<h1 id="nosql">NoSQL</h1>
<p>The sexy thing to do would be to use mongodb, couchdb or cassandra. But that migration, and writing all the map-reduce functions, plus removing activerecord and rewriting all my views - would probably take more than the 60 minutes I assigned to this problem.</p>
<p>So… What can we do with postgres?</p>
<h1 id="daily-sharding">Daily sharding</h1>
<p>The easiest solution is a <a href="http://stackoverflow.com/questions/994882/what-is-a-good-way-to-horizontal-shard-in-postgresql">variant on this</a>, where you use table inheritance to insert each days tweets into a distinct table, and then query the parent table to get an aggregate view.</p>
<p>So you just have a daily cronjob to drop any tweet tables more than 7 days old, and then create a new table for tomorrows tweets. Since tables are stored seperately on disk, the drop table is essentially free.</p>
<h1 id="performance-concerns">Performance concerns</h1>
<p>Indexes can’t be built up across multiple tables (and if you did do that, you’d need to rebuild them each day when you dropped 1/8th of the indexed data), so badly written queries could end up with a sequential scan across all tables. I’ve only had the new system running for a few hours, so I’ll know more about performance implications later this week, but since most of my queries run on only the last 24 hours, I hope I can hint postgres to query the most recent table (using the daily indexes) first and only then move back to the second most recent table.</p>
<h1 id="the-code">The code</h1>
<p>Nice and simple - a cronjob to create tomorrows table…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def self.create_tomorrows_table
date = 1.days.from_now.to_date
date_string = date.to_s.gsub(/-/,'_')
sql =<< EOF
CREATE TABLE tweets_#{date_string} (
CHECK ( DATE(created_at) = DATE('#{date}') )
) INHERITS (tweets);
EOF
ActiveRecord::Base.connection.execute sql
end
</code></pre>
</div>
<p>And then a sharded save method (note that I have a very customized tweet import job, so I don’t have to worry about tweet ids / saving associations).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def sharded_save!
date = self.created_at.to_date
date_string = date.to_s.gsub(/-/,'_')
quoted_attributes = attributes_with_quotes
sql = << EOF
INSERT INTO
tweets_#{date_string} (#{quoted_column_names.join(', ')})
VALUES
(#{quoted_attributes.values.join(', ')})
EOF
ActiveRecord::Base.connection.execute sql
end
</code></pre>
</div>
<h1 id="to-be-continued">To be continued…</h1>
<p>So this is just day one, I’ll find out more about this scheme later in the week, but I (yet again) was impressed by how nice a modern postgresql installation is. Well done the postgres team.</p>
Jquery mobile on Nokia2010-11-25T00:00:00+00:00http://bnolan.github.com/2010/11/25/jquery-mobile-on-nokia<p>My Nokia 5230 arrived today. What a cool phone for 120€! It has satellite navigation (with a suction cup to stick it on the windscreen), a big screen, decent browser and facebook right on the front page. It’s no iphone, but it’s 1/5th the price.</p>
<p>As well as using the phone to navigate our way to the some quiet beaches north of Brisbane, I’m using it to debug <a href="http://www.weheartplaces.com/">weheartplaces</a>, so I can get my app on the Ovi store.</p>
<h2 id="first-attempt">First attempt</h2>
<p>I loaded up jquerymobile.com on the nokia and was sorely disapointed, the current alpha of Jquery mobile doesn’t support Nokia S60 v5 (s60 v5 is the touch-screen interface for Nokias) browsers. I quickly set about to see how much work was needed to get things going.</p>
<h2 id="an-hour-later">An hour later</h2>
<p><img src="/images/nokiamobile.png" />
<cite>The backbone mobile demo running on a 5230</cite></p>
<p>It <a href="http://bennolan.com/science/backbone-mobile/">works</a>! I created and solved two issues on the Jquery Mobile github - one about <a href="https://github.com/jquery/jquery-mobile/issuesearch?state=open&q=history#issue/543">history</a> and another about <a href="https://github.com/jquery/jquery-mobile/issuesearch?state=open&q=screen.height#issue/542">window.innerHeight</a> - and suddenly it all started working.</p>
<p>There are a few things missing in the current S60 implementation, there is no inter-screen animation and some of the CSS looks a bit rough on the symbian browser, but it’s totally functional and looks nice enough.</p>
<h2 id="fixes-needed-to-jquery-mobile-for-s60">Fixes needed to jquery mobile for s60</h2>
<ol>
<li>Promote s60 v5 to a grade a browser</li>
<li>Fix the history tracking (solution at issue 542)</li>
<li>Fix window.innerHeight in jquery (solution at issue 542 )</li>
<li>Fix inter page animations</li>
<li>Fix png / css ugliness</li>
</ol>
<h2 id="one-code-base-at-least-three-platforms">One code base, at least three platforms</h2>
<p><img src="/images/iphonemobile.png" />
<cite>The backbone mobile demo running on iPhone</cite></p>
<p>It’s exciting to be able to write an application once, in a comfortable development environment (Safari on the desktop), then deploy the same code to iPhone, Android and Nokia. I expect start-ups will begin using this toolchain to build apps for multiple appstores at once. To try out the jquery mobile + backbone app on your Nokia (you’ll need a s60 v5 device), Android or iPhone, go to <a href="http://bennolan.com/science/backbone-mobile/">my backbone mobile demo</a>.</p>
Backbone Mobile Example2010-11-24T00:00:00+00:00http://bnolan.github.com/2010/11/24/backbone-jquery-demo<p>After my post <a href="http://bennolan.com/2010/11/23/backbone-and-jquery-mobile.html">yesterday</a> - I decided to extract out an example app from my current work.</p>
<p><img src="/images/bbmobile.png" />
<cite>A backbone mobile app running in Safari</cite></p>
<p>You can <a href="http://bennolan.com/science/backbone-mobile/">try the app</a> in your browser, or view the <a href="https://github.com/bnolan/backbone-mobile">source code on github</a>.</p>
<h1 id="technology">Technology</h1>
<p>I’m using the Foursquare API, Backbone.js, Underscore.js, CoffeeScript, jQuery and jQuery Mobile. The application is written in CoffeeScript - if you make changes to <a href="https://github.com/bnolan/Backbone-Mobile/blob/master/application.coffee">application.coffee</a> - you will need to rerun coffeescript to recompile application.js.</p>
<p>If you aren’t up to date with coffeescript you can probably read through application.js and understand most of the code. I also cached a foursquare venue API search result as foursquare.json. The foursquare API doesn’t support JSONP so I couldn’t use it directly.</p>
<h1 id="patch-to-jquery-mobile">Patch to jQuery Mobile</h1>
<p>There is only patch require to the libraries - one that I discussed in my <a href="http://bennolan.com/2010/11/23/backbone-and-jquery-mobile.html">previous post</a> to make jQuery route the URLs correctly.</p>
<h1 id="indexhtml"><a href="https://github.com/bnolan/Backbone-Mobile/blob/master/index.html">Index.html</a></h1>
<p>The index.html has very little in it, just the outline HTML for the first page to be displayed. The actual page content is generated by HomeView in application.coffee. If you haven’t seen the data-role stuff before, I recommend you read through the jquery mobile docs.</p>
<h1 id="applicationcoffee"><a href="https://github.com/bnolan/Backbone-Mobile/blob/master/application.coffee">Application.coffee</a></h1>
<p>Pretty much the entire application is written in application.coffee.</p>
<h2 id="app">app</h2>
<p>A namespace with a few helpers I use to work with jquery mobile better. <code class="highlighter-rouge">activePage</code> returns the currently displayed page. <code class="highlighter-rouge">reapplyStyles</code> make jQuery mobile convert elements tagged with data-role=”whatever” into correctly displayed jquery mobile widgets. The next version of jQuery mobile might fix the .page() method, and so reapplyStyles can be removed.</p>
<h2 id="venue-class">Venue class</h2>
<div class="highlighter-rouge"><pre class="highlight"><code>class Venue extends Backbone.Model
getName: ->
@get('name')
getAddress: ->
[@get('address'), @get('city'), @get('state')].join ", "
getImageUrl: ->
@get('photo_url')
getLatitude: ->
@get('geolat')
getLongitude: ->
@get('geolong')
getMapUrl: (width, height) ->
width ||= 300
height ||= 220
"http://maps.google.com/maps/api/staticmap?center=#{@getLatitude()},#{@getLongitude()}&zoom=14&size=#{width}x#{height}&maptype=terrain&markers=color:red|#{@getLatitude()},#{@getLongitude()}&sensor=false"
</code></pre>
</div>
<p>This is a backbone model that takes a hash of data and makes it available with a few helper methods.</p>
<h2 id="venuecollection">VenueCollection</h2>
<div class="highlighter-rouge"><pre class="highlight"><code>class VenueCollection extends Backbone.Collection
model : Venue
constructor: ->
@refresh($FOURSQUARE_JSON)
</code></pre>
</div>
<p>This is instantiated as Venues (try running Venues.models in your javascript console and you’ll see all the loaded venues). When the collection is created, I call refresh with the FOURSQUARE_JSON (which is loaded from foursquare.js) which populates the collection immediately.</p>
<h2 id="editvenueview">EditVenueView</h2>
<div class="highlighter-rouge"><pre class="highlight"><code>class EditVenueView extends Backbone.View
constructor: ->
super
# Get the active page from jquery mobile. We need to keep track of what this
# dom element is so that we can refresh the page when the page is no longer active.
@el = app.activePage()
@template = _.template('''
<form action="#venue-<%= venue.cid %>-update" method="post">
<div data-role="fieldcontain">
<label>Name</label>
<input type="text" value="<%= venue.getName() %>" name="name" />
</div>
<div data-role="fieldcontain">
<label>Address</label>
<input type="text" value="<%= venue.get('address') %>" name="address" />
</div>
<div data-role="fieldcontain">
<label>City</label>
<input type="text" value="<%= venue.get('city') %>" name="city" />
</div>
<div data-role="fieldcontain">
<label>State</label>
<input type="text" value="<%= venue.get('state') %>" name="state" />
</div>
<button type="submit" data-role="button">Save</button>
</form>
''')
# Watch for changes to the model and redraw the view
@model.bind 'change', @render
# Draw the view
@render()
events : {
"submit form" : "onSubmit"
}
onSubmit: (e) ->
@model.set {
name : @$("input[name='name']").val(),
address : @$("input[name='address']").val(),
city : @$("input[name='city']").val(),
state : @$("input[name='state']").val()
}
@model.trigger('change')
app.goBack()
e.preventDefault()
e.stopPropagation()
render: =>
# Set the name of the page
@el.find('h1').text("Editing #{@model.getName()}")
# Render the content
@el.find('.ui-content').html(@template({venue : @model}))
# A hacky way of reapplying the jquery mobile styles
app.reapplyStyles(@el)
# Delegate from the events hash
@delegateEvents()
</code></pre>
</div>
<p>The first view. It sets the active page to the @el attribute, so that it can refresh the page later on (remember that the activePage may change when the user navigates to another page, so we need to be able to refer to what element we drew the form in). Then we create a template using underscore.template. The underscore template library uses ERB style syntax to generate HTML. Remember that the ERBs are evaluated using the javascript interpreter, not coffeescript, so there’s a bit of mixing and matching of coding styles going on here.</p>
<p>We bind to the <code class="highlighter-rouge">change</code> event on the model so that if model is updated somewhere else, this form is automatically re rendered. We watch for the submit event on the form (by calling @delegateEvents), so that we can handle the form submission ourselves in <code class="highlighter-rouge">onSubmit</code>.</p>
<p><code class="highlighter-rouge">onSubmit</code> sets some properties on the model by reading them out of the inputs, then triggers <code class="highlighter-rouge">change</code> on the model. This would normally be a call to @model.save(), but we don’t have a datastore to save the data too - so we just throw away the changes. In my own apps I use localStorage to persist the changes to the data, and then have a sync method that synchronises the local store with my production servers.</p>
<p><code class="highlighter-rouge">render</code> sets the title of the page, renders the template into the .ui-content div, calls reapplyStyles so that jquery mobile can do it’s magic, and then calls delegateEvents so that the events listed in the @events hash is applied to all the created elements.</p>
<h2 id="showvenueview">ShowVenueView</h2>
<p>Simpler than the EditVenueView, except it generates some internal urls:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><a href="#venues-<%= venue.cid %>-edit">Edit</a>
</code></pre>
</div>
<p>This URL is recognized by backbone (see the routes in the controller) and will cause the edit method on the HomeController to be called. We also bind to <code class="highlighter-rouge">change</code> here as well, so that when you save changes to the form, it automatically redraws the form. This is one of the main advantages of backbone.js - you can loosely couple views and not have to think of what views need to be redrawn when you make a change to a model, it happens automatically.</p>
<h2 id="homeview">HomeView</h2>
<p>Displays the list of places. Uses the <code class="highlighter-rouge">.each()</code> method that underscore.js makes available (or delegates to the native browser implementation) to iterate over all the venues and create a link to them.</p>
<h2 id="homecontroller">HomeController</h2>
<div class="highlighter-rouge"><pre class="highlight"><code>class HomeController extends Backbone.Controller
routes :
"venues-:cid-edit" : "edit"
"venues-:cid" : "show"
"home" : "home"
constructor: ->
super
@_views = {}
home : ->
@_views['home'] ||= new HomeView
show: (cid) ->
@_views["venues-#{cid}"] ||= new ShowVenueView { model : Venues.getByCid(cid) }
edit: (cid) ->
@_views["venues-#{cid}-edit"] ||= new EditVenueView { model : Venues.getByCid(cid) }
</code></pre>
</div>
<p>I only have one controller in this app, but you could easily split it into two (a home controller and a venues controller). First up - we define the routes that we match. Note the routes have to be in an order of decreasing specificity, otherwise <code class="highlighter-rouge">venues-:cid</code> would gobble up <code class="highlighter-rouge">venues-1235-edit</code>.</p>
<p>I created <code class="highlighter-rouge">@_views</code> hash that I use to keep track of all the views that I have created, since I don’t want to generate two instances of the same view. This is quite a naive implementation, because views can be created dozens of times if the user clicked on every single Venue. We would either want to expire old views, or re-use views - but I haven’t worked out a way to manage that yet.</p>
<p>Home, show and edit are all called by the Backbone router (see Backbone.history) when the anchor part of the current url (event <code class="highlighter-rouge">hashchange</code>) changes. Each action is called with the parameters extracted from the routing hash. So all we have to do is instantiate the view and our app is ready to go.</p>
<p>Finally, when the document is ready we start the backbone router and render the initial view.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I’ve done less than a weeks development with jquery mobile (I’d done a lot more using jqTouch), but it seems like a nice complement to backbone.js. It’s fantastic that you can build an app using your current web experience, that can be sold in the Ovi Store (via Nokia Webruntime), Android Marketplace or iTunes Store (Phonegap).</p>
<p>I’ll try and keep things updated as I build out <a href="http://www.weheartplaces.com/">my own app</a>, and see how things scale up.</p>
Backbone.js and Jquery mobile2010-11-23T00:00:00+00:00http://bnolan.github.com/2010/11/23/backbone-and-jquery-mobile<p>The idea behind <a href="http://weheartplaces.com/">Weheartplaces</a> is that I want to be able to bookmark places to go on my desktop computer, and then sync these places to my mobile so I can access them offline.</p>
<p>I’ve created a json API to weheartplaces, here are my bookmarks in <a href="http://www.weheartplaces.com/users/110237153/bookmarks.json">json format</a>. I’m loading these bookmarks using <a href="http://documentcloud.github.com/backbone/">backbone.js</a> and then displaying them using <a href="http://jquerymobile.com/">jquery mobile</a>. Here’s a little breakdown of how I’m doing it:</p>
<h1 id="list-view">List view</h1>
<p>Create a list view like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><ul data-role="listview" data-inset="true" data-theme="c" data-dividertheme="b">
</ul>
</code></pre>
</div>
<p>This is automatically converted into a nice looking list view by jquery mobile. You then iterate over all the elements in your Backbone collection to add them to the list:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>for bookmark in Bookmarks.models
a = $("<a />")
a.text(bookmark.get('place').name)
a.attr 'href', "#bookmarks-#{bookmark.cid}"
a.wrap("<li />").parent().appendTo(ul)
</code></pre>
</div>
<p>Notice the url that each bookmark points to - #bookmarks-12345. I use backbone routes to recognize this URL and display the correct content.</p>
<h1 id="combining-the-jquery-and-backbone-routers">Combining the jquery and backbone routers</h1>
<p>When you click a link - jquery mobile intercepts the click and looks for a div with the same name as the anchor fragment of the url you just clicked on. So for example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><a href="#home">Home</a>
</code></pre>
</div>
<p>Will cause jquery to show this div:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><div id="home">some content...</div>
</code></pre>
</div>
<p>The problem is that in my list view, I’m linking to pages that don’t exist yet - so I have to add some code to the jquery mobile routing code to create and show an empty div.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>// This code goes in $.mobile.changePage...
if( url ){
to = $( "[id='" + url + "']" ),
fileUrl = getFileURL(url);
if(to.length == 0){ // Page could not be found
console.log("Routing to " + url + "...");
to = $("<div data-role='page' id='" + url + "'><div data-role='header'><h1>&nbsp;</h1></div><div data-role='content'><img src='images/ajax-loader.png' /></div></div>").appendTo('body')
enhancePage();
}
}
</code></pre>
</div>
<p>Jquery will display this page, and change the anchor fragment of the current URL. Backbone detects the URL change by watching for a <code class="highlighter-rouge">hashchange</code> event, and will then call your matching route - in my case:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class HomeController extends Backbone.Controller
routes :
"home" : "home"
"bookmarks-:cid" : "show"
</code></pre>
</div>
<h1 id="showing-the-bookmark">Showing the bookmark</h1>
<p>Once that magic has happened, the show method on the HomeController will be called:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>show: (cid) ->
new ShowBookmarkView { model : Bookmarks.getByCid(cid) }
</code></pre>
</div>
<p>This creates a new ShowBookmarkView, and that view generates the HTML (using underscore.js templates) that is inserted into the currently active page.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$(".ui-page-active").html(@template({bookmark : @model}))
</code></pre>
</div>
<p>If I get time I will put a small demo file up on github to show how all this hangs together - but feel free to post questions below.</p>
Back from Hiatus2010-11-22T00:00:00+00:00http://bnolan.github.com/2010/11/22/back-from-hiatus<p>Sorry I’ve neglected my blog for two weeks. I was finishing up a contract, and then took a week break to work on some personal projects, as well as sort out various life things.</p>
<h1 id="travelling-around-new-zealand">Travelling around New Zealand</h1>
<p><img src="http://www.weheartplaces.com/system/uploads/10/large/800px-Cathedral_Cove,_Coromandel.JPG?1290122997" />
<cite><a href="http://www.weheartplaces.com/users/110237153/bookmarks/702622968">Cathedral Cove</a></cite></p>
<p>In February, my girlfriend and I are travelling around New Zealand for 2 months, before heading off to Brisbane, then various other destinations around the world. To get the most out of our trip, we’ve been planning a bunch of places to go on weheartplaces. It’s still early days - but if you go to my <a href="http://www.weheartplaces.com/users/110237153/guides">guides page</a> you can see pictures and addresses for some of the campgrounds, breweries and beaches we want to visit while we’re travelling around.</p>
<p>The DOC (Department of Conservation) campsites are the best, since they cost only $6-$10 a night, and tend to be in more remote areas.</p>
<p>I had an interesting talk with some New Zealand based travel guys about making their campsite data accessible offline, so if that comes up, Rissa and I should be able to travel around with a full database of every nice freedom or DOC campsite available.</p>
<h1 id="my-favourite-places-in-the-world">My favourite places in the world.</h1>
<p>I made a quick list of my <a href="http://www.weheartplaces.com/users/110237153/guides/5">Favourite places in the world</a>:</p>
<p><a href="http://www.weheartplaces.com/users/110237153/bookmarks/702622926">Midnight Espresso</a>, <a href="http://www.weheartplaces.com/users/110237153/bookmarks/702622967">Hofbraükeller</a>, <a href="http://www.weheartplaces.com/users/110237153/bookmarks/702622968">Cathedral Cove</a>, and <a href="http://www.weheartplaces.com/users/110237153/bookmarks/702622969">Ferchensee</a>.</p>
<h1 id="nokia-5230">Nokia 5230</h1>
<p>I bought another new Nokia, this time a Nokia 5230, which has the more modern Web Runtime 7.2, so that I can build some backbone.js applications that run on IOS, Android or Nokia.</p>
<p>Compared to some of my other projects, which are pure ruby on rails, no javascript at all - my Phonegap / Web Runtime apps have a gnarly technology stack:</p>
<ul>
<li>Coffeescript</li>
<li>Backbone.js</li>
<li>Jquery mobile</li>
<li>Less CSS</li>
<li>Qunit</li>
<li>Phonegap</li>
</ul>
<p>I’ll talk more about phonegap and backbone.js + jquery mobile later this week.</p>
Level Interaction2010-11-04T00:00:00+00:00http://bnolan.github.com/2010/11/04/level-interaction<p>I’ve just posted a <a href="http://www.youtube.com/watch?v=YWzGjUiL4LY">new video</a> of Hutch. New features include terrain generation, better collision detection, item interaction, tile interaction. I’ve been looking around the web and also found a few people doing truly amazing javascript stuff:</p>
<ul>
<li><a href="http://rumpetroll.com/">Rumpetroll</a></li>
<li><a href="http://rocketpack.fi/">Rocketpack</a></li>
</ul>
<p>It really looks like Javascript gaming is about to take off!</p>
Game design2010-11-03T00:00:00+00:00http://bnolan.github.com/2010/11/03/game-design<p>I’ve been playing around with <a href="http://www.youtube.com/watch?v=BS27XGxwMes">Hutch</a> lately, and have been reading some game design articles. I found this <a href="http://www.graftgold.com/">pearl of wisdom</a> from the developer of some old Commodore 64 games that I used to love as a kid.</p>
<p>(As an aside, said C64 developer now writes insurance software because it pays better).</p>
<p>The most interesting part of the article I thought was the developers game design process:</p>
<ul>
<li>We investigated the platform and designed and wrote graphics architecture to exploit some feature of the machine.</li>
<li>We considered game environments that utilised that feature.</li>
<li>We create a player in a prototype level and worked on the control and movement.</li>
<li>We decided on the game idea and added enemies to the test level.</li>
<li>We worked on real levels constantly coming up with new ideas as we progressed.</li>
<li>We reviewed the levels , adding new ideas in where necessary</li>
</ul>
<p>This is very similar to how I’ve been working on Hutch. Invent the graphics engine, think up game environments you can render in realtime, add player movement and control. The next stage is to add items and enemies.</p>
<p>Hopefully I’ll get time to do that later in the week.</p>
Twitterplaces2010-10-28T00:00:00+00:00http://bnolan.github.com/2010/10/28/twitterplaces<p>I went to a R users group in melbourne and got all turned on by data analysis, so I went home and started crawling all the tweets in the world and sticking them into a Postgres database. I then went and build this <a href="http://www.twitterplaces.com/">site</a>.</p>
<p>I keep playing around with the site and adding random features and new pages, but I’d like some feedback on what I could do so that people actually found the site useful. It’s cool building a site for myself, but it’d be cool if more than 10 people a day used the site. What do you lot think would be some concrete features to add to the site?</p>
<p>Here’s some of the current pages that I think are interesting but could do with some work:</p>
<ul>
<li>http://www.twitterplaces.com/muc</li>
<li>http://www.twitterplaces.com/users/dens</li>
<li>http://www.twitterplaces.com/sfo/twitter-hq</li>
<li>http://www.twitterplaces.com/sfo/mission</li>
</ul>
<p>Some technical infos:</p>
<p>I crawl the data using the streaming API, all geotagged tweets in the world (about 2m/day). The crawler is written in ruby and does a bunch of caching so it doesn’t have to hit up postgres too much.</p>
<p>I use postgis so I can lookup bounding neighbourhoods and cities for points based on lat / long.</p>
<p>Some of the queries are insanely slow (eg which users travel the world the most), so I need to work out a better way of generating results.</p>
<p>It’s running on a AMD X4 box with 4gb of ram, so seems to keep chugging along.</p>
<p>I use varnish on the front of the site to cache html for a few minutes after each visit.</p>
<p>TLDR: Have <a href="http://www.twitterplaces.com/">site</a>, what features should I add?</p>
Additions to Backbone views2010-10-27T00:00:00+00:00http://bnolan.github.com/2010/10/27/additions-to-backbone-views<p>I took Matt McCrays excellent <a href="http://mattmccray.blogspot.com/2010/10/using-backbonejs-in-coffeescript.html?spref=tw">Coffeescript glue</a> for Backbone.js and moved my <code class="highlighter-rouge">saveChanges</code> function into it.</p>
<p>The function serializes all the form elements in the rendered elements and updates the associated model. You can use this code by binding to ‘submit form’ using <code class="highlighter-rouge">@handleEvents</code> and calling saveChanges and suppressing the default form submission.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class View
constructor: ->
Backbone.View.apply(this, arguments)
saveChanges: ->
properties = {}
for input in @el.find('input,textarea')
value = $(input).val()
properties[input.name] = value
@model.set properties
@model.save()
app.saving()
_.extend(View::, Backbone.View.prototype)
</code></pre>
</div>
<p>A small example from <a href="http://weheartplaces.com/">Weheartplaces</a>…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class TripInspector extends App.View
constructor: ->
super
@template= JST["trip_inspector/show"]
@render() if @model?
render: ->
@el.html(@template { trip : @model })
@handleEvents {
'submit form' : 'submit'
}
submit: (e) =>
e.preventDefault()
@saveChanges()
this.TripInspector = TripInspector
</code></pre>
</div>
Tagging2010-10-26T00:00:00+00:00http://bnolan.github.com/2010/10/26/tagging<p>I’ve been doing some work on getting the private beta of <a href="http://weheartplaces.com/">Weheartplaces</a> out. I’m using tagging as a way to categorize places, so you can quickly add a place and tag as ‘todo’, ‘cafe’ or possibly ‘avoid’.</p>
<p><img src="/images/taggingplaces.png" />
<cite>Places tagging interface for Weheartplaces</cite></p>
<p>I found <a href="http://github.com/rafudu/Tagbox">tagbox</a> by Rafudu still works okay with current jquery. It’s a bit creaky, but with some CSS tweaks it worked okay out of the box.</p>
Gisborne2010-10-22T00:00:00+00:00http://bnolan.github.com/2010/10/22/gisborne<p>I’ve been away from the internet for a bit - chewing through some work for the good folks at <a href="http://youdo.co.nz/">Youdo</a> (you should try their ITIL tool - <a href="http://beetil.com/">Beetil</a>) and doing various business development work. One of the most enjoyable business trips I’ve ever been on can be summarised by this photo:</p>
<p><img src="/images/rankers.jpg" />
<cite>A remote beach on the east cape of NZ</cite></p>
Four for four2010-10-15T00:00:00+00:00http://bnolan.github.com/2010/10/15/four-for-four<p>I just sat down at my Mac. Here’s a recap of the last hour.</p>
<ol>
<li>Thirty minute bike ride on muddy singletrain</li>
<li>It stopped raining and the sun came out</li>
<li>As I got out of the shower my girlfriend rang up to ask what I wanted her to bring for lunch</li>
<li>I pressed play on iTunes and got some pimping <a href="http://www.youtube.com/watch?v=pOjw9rAWt2E">Murs</a></li>
</ol>
<p>Yeeeeeeeeeeah.</p>
A modern webapp2010-10-14T00:00:00+00:00http://bnolan.github.com/2010/10/14/a-modern-webapp<p>Want to write a modern webapp using lots of javascript? Here’s my recommended toolkit:</p>
<ul>
<li><a href="http://jashkenas.github.com/coffee-script/">CoffeeScript</a> for brevity</li>
<li><a href="http://jquery.com/">Jquery</a> for sanity</li>
<li><a href="http://jqueryui.com/">Jquery UI</a> (with <a href="http://taitems.tumblr.com/post/482577430/introducing-aristo-a-jquery-ui-theme">Aristo</a>) for your design and widgets</li>
<li><a href="http://documentcloud.github.com/underscore/">Underscore.js</a> for your views (see _.template)</li>
<li><a href="http://documentcloud.github.com/backbone/">Backbone.js</a> for your models andd controllerts</li>
<li><a href="http://documentcloud.github.com/jammit/">Jammit</a> for asset packaging</li>
</ul>
<p>I’ve only recently discovered backbone and jammit, but I have a lot of respect for <a href="http://github.com/jashkenas/">Jeremy</a> so they’re my <a href="http://weheartplaces.com/">bet</a>.</p>
Splatter2010-10-12T00:00:00+00:00http://bnolan.github.com/2010/10/12/splatter<p>After watching <a href="http://www.banksyfilm.com/">Exit Through The Gift Shop</a> last night, I wanted to do something creative, so I played around with isometric projection using css3 and applying stencils to canvas. I came up with something vaguely similair to what I was imagining….</p>
<p><img src="/images/splatter.png" />
<cite>Designing a little world</cite></p>
My jqtouch fork2010-10-08T00:00:00+00:00http://bnolan.github.com/2010/10/08/my-jqtouch-fork<p>Sorry for not blogging this week. I’ve been in a client office and furiously working on getting a sprint out. On Monday I had my product development day, so I started work on <a href="http://weheartplaces.com/">weheartplaces</a>. The first iteration of WHP is due out by the end of October, and I plan to have it working on Iphone, Nokia and Android. To do this I’m using <a href="http://blog.jqtouch.com/">jqtouch</a> which hasn’t seen a lot of work in the past year.</p>
<p>I forked jqtouch onto my <a href="http://github.com/bnolan/">github</a> and have been pushing a few changes. They aren’t all ready for the primetime yet - but the main design changes I’ll be making:</p>
<ul>
<li>URL routing scheme, so you can have controllers and actions to handle requests</li>
<li>Use pushState to have proper urls (instead of #location urls)</li>
<li>Specced and tested using <a href="http://github.com/pivotal/jasmine">Jasmine</a></li>
<li>Better form support</li>
</ul>
<p>I also have some ideas about how I can make an analog to jqtouch that will work with the Web Runtime that Nokia supports for development on their phones. I’m doing my jQTouch modifications in pure javascript, but the Weheartplaces application is written using Coffeescript.</p>
Silverstripe hosting2010-10-05T00:00:00+00:00http://bnolan.github.com/2010/10/05/silverstripe-hosting<p>I have a little joke going that I think a hosted <a href="http://silverstripe.com/">silverstripe</a> would be quite a good competitor to wordpress.</p>
<p><img src="/images/brownstripe.png" />
<cite>The silverstripe admin interface with a few tweaks</cite></p>
<p>Silverstripe is an excellent CMS built by some of my friends in Wellington. They have a big company now, over 30 people, lots of big clients - and they even have a product - <a href="http://silverstripe.com/dawn/">Dawn</a>. Wordpress.com is the free hosting provider that runs <a href="http://mu.wordpress.org/">Wordpress MU</a> and has approximately a <a href="http://ma.tt/2010/09/msn-spaces-closing-becomes-wp-com/#comments">bajillion</a> blogs on it.</p>
<p>The revenue stream is that on wp.com, you have to pay to get <a href="http://en.wordpress.com/products/">premium features</a>, like CSS editing, premium themes, custom domain names. I think it’s an awesome business model that the Silverstripe team should take a look at.</p>
A preview of new design2010-10-01T00:00:00+00:00http://bnolan.github.com/2010/10/01/new-menu<p>I have a new menu and header design coming for twitterplaces. A preview:</p>
<p><img src="/images/previewmenu.png" /></p>
Random Walk2010-09-30T00:00:00+00:00http://bnolan.github.com/2010/09/30/random-walk<p>For <a href="http://twitterplaces.com/">twitterplaces</a>, I’ve been using the Random Walk theory of software development. I pick up the software each day, and work on whatever feature I feel like. There’s no plan, no vision, no long term overarching story that directs my work.</p>
<p>I just go to a page that looks scruffier than the rest, then tidy it up. Or play with the data and find another interesting report, then put that in the site somewhere. Occasionally I may have to do a bit of architecture or scaling work, but i don’t plan for it - I just do it when required. And I’m always pushing to make the code easily understandable. A random walk doesn’t mean spaghetti code.</p>
<p>It just means, don’t beat on feature “a” until it’s done. Work on a,b,c and invent feature d. That way you won’t wear out and you won’t become uninspired.</p>
<p>Sure this won’t work for your on-spec, on deadline project, but for your own projects, or experimentation in user interface design, just random walk through your project.</p>
<p>You may be surprised where you end up.</p>
<p><img src="/images/graphviz.png" />
<cite>A graphviz plot of adjacent commits on twitterplaces</cite></p>
Against Me!2010-09-29T00:00:00+00:00http://bnolan.github.com/2010/09/29/against-me<p>My favourite band in the world cancelled their Australian and New Zealand tour.</p>
<object width="480" height="385"><param name="movie" value="http://www.youtube.com/v/mAkSIJ9BSi8?fs=1&hl=en_US" /></param><param name="allowFullScreen" value="true" /></param><param name="allowscriptaccess" value="always" /></param><embed src="http://www.youtube.com/v/mAkSIJ9BSi8?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385" /></embed></object>
<p>Sad face.</p>
The Art of Happiness2010-09-28T00:00:00+00:00http://bnolan.github.com/2010/09/28/the-art-of-happiness<p>I think this sums it up for me:</p>
<quote>
When you feel lacking in happiness.
Don't delay feeling happy
for when some future
achievement has happened.
Let yourself feel happy now.
</quote>
<p><a href="http://en.wikipedia.org/wiki/Deferred_gratification">Delayed gratification</a> doesn’t apply to happiness.</p>
An island of Javascript2010-09-27T00:00:00+00:00http://bnolan.github.com/2010/09/27/three.js<p>I ready about the sale of <a href="http://www.dextrose.com/">Dextrose</a> to Zynga by <a href="http://twitter.com/pbakaus">Paul Baukaus</a>, and it got me motivated to poke around with some html5 gaming again (a nice sunday morning thing to do). The outcome was an island generator using <a href="http://github.com/mrdoob/three.js/">three.js</a> and jquery ui.</p>
<p><img src="/images/island-zing.png" />
<cite>Rudimentary but functional. 3d javascript for the future!</cite></p>
<p>When you start the demo, you just get an empty sea, but move the cube around with the arrow keys and hold down space to raise some mountains out of the sea. The demo has a bad case of z-fighting, but I only spent an hour or so on this, so I’ll look into those issues in the future sometime.</p>
<h1 id="threejs">Three.js</h1>
<p>Three is a great library from one of the authors of papervision 3d (so they know what they’re doing when it comes to browser-based graphics). It is surprisingly fast on modern browsers - the slowest thing seems to be the canvas drawing calls, which aren’t hardware accelerated. The great thing about doing javascript work though, is that browsers are being updated so often at the moment (Safari, IE9, Firefox and Chrome) that what is slow but workable today, will be liquid smooth in 6 months time (assuming browser development keeps going like this).</p>
<p>Three also has output support for canvas and webgl. I’m using canvas in this demo.</p>
<h1 id="a-3d-game-engine">A 3d game engine</h1>
<p>I’d like to play with this demo some more, and turn it into a mini game. I’m currently thinking of using three to render to multiple canvases, and position the camera so that the terrain doesn’t obscure itself, and I can use the browsers native z-order stacking for elements on top of the map. This should make a realtime 3d world doable without pegging the CPU (I have a strong aversion to pinning the CPU to draw a 400 face terrain when this macbook can play TF2 at native resolution).</p>
<p>Then add multiplayer via websockets and node.js and you’d have that multiplayer canvas game that Simon Willison <a href="http://simonwillison.net/2010/Feb/8/pseudo/#comments">predicted</a>.</p>
New Twitter Analyzed2010-09-24T00:00:00+00:00http://bnolan.github.com/2010/09/24/new-twitter<p>I was interested to read the twitter engineering teams <a href="http://engineering.twitter.com/2010/09/tech-behind-new-twittercom.html">overview</a> of how the newtwitter works.</p>
<p>This is the kind of app that is becoming more and more common, as it becomes viable to build apps in pure javascript, with just a json API as the backend. There are downsides to this kind of development, mostly in the inability for google to index the site - but for an app like twitter which are indexed by google using the pubsub interface (afaik), there’s no reason not to provide a pure javascript / html5 interface for modern browsers.</p>
<p>I haven’t got new twitter yet - so I hit up a friend and got him to send me the URLs for the new twitter javascript. I guessed the pre-closure URLS, tidied them up using html tidy and textmate - and we have these:</p>
<ul>
<li><a href="/science/newtwitter/base.txt">base.js</a></li>
<li><a href="/science/newtwitter/phoenix.txt">phoenix.js</a></li>
</ul>
<h1 id="basejs">Base.js</h1>
<p>Base.js seems to contain their extensions to jquery to create the twitter ‘framework’, eg:</p>
<ul>
<li>Logging</li>
<li>Debugging</li>
<li>Stack traces</li>
<li>Browser detection</li>
<li>Analytics</li>
<li>Page state / history</li>
<li>Load templates</li>
<li>Form validation</li>
<li>Help</li>
<li>Autocomplete</li>
</ul>
<p>It also has some code that uses the anywhere api (I think), to find a user by screenname. It has a bunch of code that has either been handrolled, or gathered together from different jquery plugins (eg timeAgo, numberWithDelimiter).</p>
<p>The really interesting stuff is in phoenix.js though.</p>
<h1 id="phoenixjs">Phoenix.js</h1>
<p>Newtwitter uses <a href="http://mustache.github.com/">Mustache</a> for their templates. Many of these templates are included in phoenix.js, as twttr.templates. The templates are internationalized and use classes liberally - eg <code class="highlighter-rouge">stream-tab-following</code>, <code class="highlighter-rouge">stream-tab-pending</code>, <code class="highlighter-rouge">delete-saved-search</code>.</p>
<p>If you search on any of these classnames - you’ll find code later in phoenix.js that attaches the appropriate event handler to the html generated by mustache. For example - this fragment:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><div class='delete-saved-search'>
\n <a href='#' class='delete-saved-search-x'></a>\n <a href='#'>Remove saved search</a>\n
</div>",
</code></pre>
</div>
<p>Has behaviour attached to it like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>click({
".delete-saved-search": function(B, A) {
twttr.dialogs.deleteSavedSearch(twttr.bind(this,
// ....
B.stopPropagation();
B.preventDefault();
return false
})
</code></pre>
</div>
<p>I’ve munged the code up a bit here, but you get the basic idea. Phoenix.js includes jquery (the latest release, 1.4.2), Jquery UI (1.8.4), <a href="http://www.modernizr.com/">Modernizer</a> (a library for detecting browser support for various html5 functionality), <a href="http://benalman.com/code/test/js-linkify/">Linkify</a>.</p>
<p>This was just a quick analysis of the newtwitter codebase - but I found all the mentions of <code class="highlighter-rouge">twttr.view</code> to be interesting - it seems like the twitter engineering team have built quite a strong framework for javascript apps - I wonder if they’ll release their MVC framework for other developers to create similair apps?</p>
Varnish and Rails2010-09-23T00:00:00+00:00http://bnolan.github.com/2010/09/23/varnish-and-rails<p>I got my R scripts running on production for <a href="http://twitterplaces.com/">twitterplaces</a> - so if you browse to <a href="http://twitterplaces.com/wlg">Wellington</a> or <a href="http://twitterplaces.com/lon">London</a>, you can see twitter densities. That’s all well and good and very rough, there’s a lot of work to do:</p>
<ol>
<li>The time slider looks rubbish</li>
<li>The tweet frequency histogram isn’t adjusted for the cities time offset</li>
<li>The extents of the different cities are different, so you get elongated contours</li>
<li>The contour needs to be subdivided for smoothness</li>
<li>It would be good to weigh against multiple tweets from a user</li>
</ol>
<p>But the first problem, is that the contours are exceptionally expensive to convert to json, which is done on every page load by ruby. I wanted to have an easy way to cache a page for 60 seconds, so that if the page gets ‘seagulled’ (a thousand people follow a link from techcrunch, look at the page for a second, then go back to techcrunch to comment ‘it doesnt scale get nosql why not use php’) the site doesn’t go down.</p>
<h1 id="ask-koz">Ask Koz</h1>
<p>I originally was going to do some apache rewriterule hax, but first I asked <a href="http://twitter.com/nzkoz">@nzkoz</a>, who told me to look into <a href="http://www.varnish-cache.org/">Varnish</a>. Varnish looked good so I gave it a go.</p>
<h1 id="varnish">Varnish</h1>
<p>I followed the installation instructions <a href="http://www.varnish-cache.org/installation/ubuntu">for Ubuntu</a>, then edited <code class="highlighter-rouge">/etc/defaults/varnish</code> to make varnish listen on port 80.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>DAEMON_OPTS="-a :80\
-T localhost:6082 \
</code></pre>
</div>
<p>I edited <code class="highlighter-rouge">/etc/apache2/ports.conf</code> to move apache over to port 8080:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>NameVirtualHost *:8080
Listen 8080
</code></pre>
</div>
<p>And edited the appropriate <code class="highlighter-rouge">sites-available</code> scripts to move the virtual hosts over to the new port. Then I edited my <code class="highlighter-rouge">/etc/varnish/default.vcl</code> (make sure you install Varnish 2.x, not the 1.x that comes with some old varieties of Ubuntu) like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sub vcl_recv {
unset req.http.Cookie;
}
sub vcl_fetch {
if (req.request == "GET") {
unset beresp.http.Set-Cookie;
set beresp.cacheable = true;
set beresp.ttl = 60s;
}
if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets"){
set beresp.ttl = 15m;
}
}
</code></pre>
</div>
<p>All GETs are cached for 60 seconds. Javascripts, images and stylesheets are cached (on the client side) for 15 minutes.</p>
<p>Note that this varnish config will annihilate any site that uses cookies - but twitterplaces (and weheartplaces) currently don’t use cookies at all - so everyones looks at the same pages. I then restarted varnish and <em>voila</em>, we have a fast cached site that can cope with some seagulling.</p>
<p>To check that things are working - I tailed my rails log, hit the site up with <code class="highlighter-rouge">curl -I http://twitterplaces.com/wlg/</code> and each request was served correctly, but didn’t show up in the rails logs until 60 seconds had elapsed. Varnish is very clever and will serve the old version of the cached page until the new version has finished regenerating. Varnish really does seem like a neat bit of kit.</p>
<p>Elapsed time, from learning of Varnish, to installing, configuring, testing and writing this blog post - 45 minutes.</p>
Moving to production2010-09-22T00:00:00+00:00http://bnolan.github.com/2010/09/22/r-on-ubuntu<p>I lied. I wasn’t really building twitterplaces, I just got back from 5 days snowboarding, relaxing and reading books by the fire. Now i’m back in the office and started the week with getting my production box set up so I can deploy the next release of <a href="http://twitterplaces.com/">twitterplaces</a>.</p>
<h1 id="monit">Monit</h1>
<p>I set up a monit task to run my tweet importer and restart the process if it dies. The monit task looks like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>check process importer
with pidfile /tmp/importer.pid
start program = "/usr/bin/sudo -u ben /var/www/tp/script/runner -e production Tweet.import" with timeout 20 seconds
stop program = "/bin/kill -TERM `cat /tmp/importer.pid`"
</code></pre>
</div>
<p>Note that this job requires monit 5 since it uses the <code class="highlighter-rouge">with timeout</code> parameter. I also added this line to my importer script so that it spits out the process id.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>`echo #{Process.pid} > /tmp/importer.pid`
</code></pre>
</div>
<h1 id="r-and-odbc">R and ODBC</h1>
<p>For my R scripts, I’ve been using the postgres adapter so that I can query directly from within R. It turns out the package I was using on the desktop wasn’t available on ubuntu, so I followed the instructions <a href="http://ubuntuforums.org/showthread.php?t=614715">here</a> and <a href="http://simon.bonners.ca/blog///blog5.php/2010/03/13/accessing-a-postgresql-database-from-r-using-rodbc">here</a> to get <code class="highlighter-rouge">RODBC</code> set up with postgres.</p>
<h1 id="future-tasks">Future tasks</h1>
<p>I have a few things left to do before the new release is complete and ready for people to use:</p>
<ol>
<li><strong>Data massaging</strong> - the production database needs massaging and normalization of the bounding boxes for cities - I’ve been doing this adhoc, but it needs to be integrated into the twitter import process.</li>
<li><strong>Data analysis cronjobs</strong> - some of the data analysis jobs are best run hourly (for example the contour map generation), so these need to be scheduled as a cronjob</li>
<li><strong>Postgres partitioning</strong> - twitterplaces now imports over 2m tweets a day. To reduce the load on postgres, I’m going to use <a href="http://www.postgresql.org/docs/current/static/ddl-partitioning.html">partitioning</a> to reduce the size of the indices and quicker data truncation (I can just drop week old tables, instead of a delete and vacuum).</li>
</ol>
<p>Hopefully i’ll be able to get some of that out later this week.</p>
A break in transmission2010-09-16T00:00:00+00:00http://bnolan.github.com/2010/09/16/regular-transmissions<p>I’m taking a week off blogging so I can focus on <b>building</b>. Please excuse my silence - and I look forward to resuming communications with you all sometime next week.</p>
<p><img src="http://farm5.static.flickr.com/4133/4990723483_c1a4f97d5b.jpg" /></p>
<p><cite>Manners mall redevelopment</cite></p>
Twitterplaces Next Generation2010-09-15T00:00:00+00:00http://bnolan.github.com/2010/09/15/twitterplaces-ng<p>Ever since I <a href="http://bennolan.com/2010/09/13/a-world-of-tweets.html">discovered</a> that you can index all of the worlds geotagged tweets with Twitter, I’ve been working on the next generation of twitterplaces. It’s a challenge. The app has to work on datasets the size of Wellington (around 110 tweets per day) to New York (around 15k tweets per day) and provide an intimate experience for every user.</p>
<p>One of the biggest problems with displaying lots of data on a map is that you end up with massive clouds of google markers all over the map. These look okay when there’s less than 5 markers, but above that, clicking on many markers is a bad user experience, it’s hard to get an aggregate view of what’s going on.</p>
<p><img src="/images/twng.png" />
<cite>Viewing wellington tweets grouped by contour density maps</cite></p>
<p>I’ve already posted about using R to create contour maps of tweet density. I’ve now taken that code and changed it from a manual job, into something automated. It’s not production ready yet - but the idea is:</p>
<ol>
<li>Contour generator (R script) runs hourly on top 1000 cities</li>
<li>R queries postgres for tweets in the last hour</li>
<li>Generate the density matrix, and generate contour lines</li>
<li>Convert contours into <a href="http://postgis.refractions.net/docs/ch04.html#RefObject">WKT</a> polygons and save to the database</li>
</ol>
<p>When you display a city page - you export the contour polygons into a json array (indexed by hour) and then give the user some UI for changing what time window they are looking at.</p>
<h1 id="realtime-contour-generation">Realtime contour generation</h1>
<p>I’m not in a hurry to try and do this in realtime, but it has been at the back of my mind. My bookmarks tagged <a href="http://www.delicious.com/benn/contour">contour</a> have my notes on the basic algorithm. Any query that filters keyword, user_id, generate contour maps of an individual suburb, do language analysis, sentiment analysis, etc - would generate data suitable for converting into contour maps.</p>
<p>If I wanted to be able to generate this data in realtime, there are two ways I can see of doing it.</p>
<ol>
<li>Farm the raw tweet data off to the browser and generate the contour lines in Javascript</li>
<li>An optimized C++ app that uses sse / simd to generate the contour lines in a fraction of the time of R</li>
</ol>
<p>Or - I could charge customers monthly, fire up another ec2 instance and keep using the existing queries in R.</p>
Panasonic Lumix and the Optimal Prime2010-09-14T00:00:00+00:00http://bnolan.github.com/2010/09/14/panasonic-lumix<p>I’ve been waiting to make this post for a few weeks now. I bought a Pansonic Lumix GF1 with the 20mm prime (non-zoomable) lens. It’s a real beauty! It takes lovely photos, is a joy to use (nice and fast) and fits in my coat pocket. I bought it as a present to myself for three months working for Lonely Planet in Melbourne.</p>
<h1 id="choosing-the-gf1">Choosing the GF1</h1>
<p>I got interested in a camera with a nice prime lens after seeing all <a href="http://www.flickr.com/photos/maddiontour">Maddis</a> epic photos. He uses a sony Alpha and the minolta beercan lens. I started researching and quickly became interested in getting an <a href="http://www.photographyblog.com/reviews/sony_a330_review/">a330</a> and hunting down some 80s lenses - but as I read around the net about small form factor dslrs, I kept seeing micro four thirds being mentioned, in particular the GF1.</p>
<p>This <a href="http://www.dpreview.com/reviews/PanasonicGF1/">review</a> at dpreview and <a href="http://craigmod.com/journal/gf1-fieldtest/">craigmods</a> gf1 review were the articles that really sold me on the GF1. It’ll be fun to pack up our house into a big box and then go travel the world with this camera hanging off my neck.</p>
<p><img src="http://farm5.static.flickr.com/4145/4985753541_25ab2b35d9_z.jpg" />
<cite>Sweet Mothers Kitchen. Taken with the 20mm pancake lens</cite></p>
<p>See more photos from last night on this <a href="http://www.flickr.com/photos/benn/sets/72157624946265944/with/4985753541/">flickr set</a>.</p>
A world of tweets2010-09-13T00:00:00+00:00http://bnolan.github.com/2010/09/13/a-world-of-tweets<p>The Twitter streaming API now lets you recieve all geotagged tweets, not just geotagged tweets in specified areas. This means that with a command like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -d "locations=-180,-90,180,90" http://stream.twitter.com/1/statuses/filter.json -uuser:pass
</code></pre>
</div>
<p>You can recieve all the geotagged tweets twitter has. I updated my crawler to run against this complete dataset, fixed a few lingering bugs, and then let my crawler run for a few hours:</p>
<p><img src="/images/ooo.png" />
<cite>o’s indicate exact location, .’s indicate neighbourhood</cite></p>
<p>I then exported my tweets, and plotted x vs y in R:</p>
<p><img src="/images/worldtweets.png" />
<cite>plot(tweets$x, tweets$y); world(add=TRUE)</cite></p>
<p>You can clearly see a world map forming, with a great concentration of geotagged tweets in north america, europe, japan and indonesia, as well as the outlines of Australia and New Zealand.</p>
<p>What if we cluster the points using <code class="highlighter-rouge">kmeans</code>?</p>
<p><img src="/images/kmeans.png" />
<cite>Clustered into 5 groups using kmeans squared</cite></p>
<p>We get 5 clusters, western us, eastern us, europe, asia and south america. Looks like Texas is an eastern state then. It’s kind of scary that Africa is so sparse that it gets gobbled up by Europe, South America and Asia.</p>
<p>I’m keen to see what a histogram of seconds from midnight (what I call epoch) and longitude will look like - but i’ll have to wait til I have 24 hours of data. It looks like I’ll end up with about 2 million tweets per 24 hour period, which should make a good dataset.</p>
Contour maps of tweet density2010-09-10T00:00:00+00:00http://bnolan.github.com/2010/09/10/contour-maps<p>I regenerated by tweet density graphs using contour lines, and got this nice looking animation of tweet density in london over the course of a day.</p>
<object width="400" height="400"><param name="movie" value="http://www.youtube.com/v/XMNg1Hjd6qE?fs=1&hl=en_US" /></param><param name="allowFullScreen" value="true" /></param><param name="allowscriptaccess" value="always" /></param><embed src="http://www.youtube.com/v/XMNg1Hjd6qE?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="400" height="400" /></embed></object>
<p><br />
<br /></p>
<p><cite>24 hours of tweets in London</cite></p>
<p><br /></p>
<p>The important improvements:</p>
<h2 id="use-a-sliding-time-window">Use a sliding time window</h2>
<p>Before I was using hourly updates, which meant that the updates were quite abrupt, and I could have a maximum of 24 frames per movie. I used this subquery to pull out the number of seconds since midnight, so that I could use a sliding time window of one hour in R:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>extract(epoch from created_at::time)
</code></pre>
</div>
<h2 id="use-contour-lines">Use contour lines</h2>
<p>The <code class="highlighter-rouge">contour</code> function in R generates really nice contours from heightmaps. They look nice, and are easily georeferenced.</p>
<h2 id="resample-the-density-matrix">Resample the density matrix</h2>
<p>I sample the tweets in a 50x50 matrix. This works, but gives quite sharp contour lines. The <code class="highlighter-rouge">image.smooth</code> function (from the image library) came in handy here, as it uses a gaussian smoothing kernel on each element in the matrix. The default settings gives a nice smooth heightfield which can be plotted by contour</p>
<h2 id="next-steps">Next steps</h2>
<p>After this - I worked out that I could call <code class="highlighter-rouge">contourLines</code> to get the line coordinates. Then by converting this to json using <code class="highlighter-rouge">rjson</code>, I plotted the contourlines onto a google map using polygons. Finally, I made it so that you can click on the polygons, and see what tweets and neighbourhoods are causing weird spikes of activity.</p>
<p>I’m busy with client work at the moment, so hopefully sometime next week I can push an update to <a href="http://twitterplaces.com/">twitterplaces</a> that features these contour maps.</p>
Imperial March2010-09-08T00:00:00+00:00http://bnolan.github.com/2010/09/08/imperial-march<p>One of the benefits from working at home is that I get to play trumpet and guitar as much as I want. Which means lots of time to practise.</p>
<object width="400" height="320"><param name="movie" value="http://www.youtube.com/v/ZiXNUzFYvmw?hl=en&fs=1" /></param><param name="allowFullScreen" value="true" /></param><param name="allowscriptaccess" value="always" /></param><embed src="http://www.youtube.com/v/ZiXNUzFYvmw?hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="400" height="320" /></embed></object>
<p><br />
<br /></p>
<p>For some reason, practising lots of the trumpet isn’t helping me get better at all.</p>
Tweet map in R2010-09-07T00:00:00+00:00http://bnolan.github.com/2010/09/07/tweet-map-in-r<p>Just as a quick follow up to yesterdays post - this is the very rough visualization I came up with using R, openGL and ffmpeg.</p>
<object width="400" height="320"><param name="movie" value="http://www.youtube.com/v/iqz6skQLkaQ?fs=1&hl=en_US" /></param><param name="allowFullScreen" value="true" /></param><param name="allowscriptaccess" value="always" /></param><embed src="http://www.youtube.com/v/iqz6skQLkaQ?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="400" height="320" /></embed></object>
<p><br />
<br /></p>
<p>The r script used was something like this. The texture isn’t correctly georeferenced, and needs to be flipped 180°, but this was as far as I got before having to move onto other work.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ourplot <- function (h) {
north = 51.6
south = 51.4
east = 0.05
west = -0.25
df <- subset(tweets, x > west & x < east & y < north & y > south & hour == h)
df <- rbind(df, data.frame(x=east, y=north, hour=-1))
df <- rbind(df, data.frame(x=west, y=south, hour=-1))
hdf <- hist2d(df$x, df$y, nbins=50, show=FALSE)
persp3d(
hdf$x,
hdf$y,
hdf$counts,
ticktype="detailed",
xlab="Longitude",
ylab="Latitude",
zlab="# Tweets",
zlim=c(-30,50),
expand=0.5,
shade=1.0,
specular="white",
col="white",
texture="/tmp/london.png",
box=FALSE,
axes=FALSE
)
rgl.snapshot(paste("/tmp/london-3d-",h,".png",sep=""))
}
for (hour in seq(0,24,by=1)) ourplot(hour)
</code></pre>
</div>
<p>I have a vague plan of taking the histogram matrix, exporting it to json and using protovis to redraw the graphs, possibly as a <a href="http://vis.stanford.edu/protovis/ex/stream.html">streamgraph</a>.</p>
Plotting Choropleths with r2010-09-06T00:00:00+00:00http://bnolan.github.com/2010/09/06/plotting-choropleths-with-r<p>Choropleths are those <a href="http://flowingdata.com/2009/11/12/how-to-make-a-us-county-thematic-map-using-free-tools/">heatmap style</a> maps you often see. I’ve been keen to render some heatmaps out of the <a href="http://twitterplaces.com">twitterplaces</a> data I have, so I spent today learning R and using it to generate some graphs.</p>
<p>First up - I installed R using the os x binaries, then went right on to install rgdal.</p>
<h2 id="installing-rgdal-on-snow-leopard">Installing rgdal on snow leopard</h2>
<p>To load map data, you first need to install the PROJ and <a href="http://www.kyngchaos.com/software:postgres">GDAL frameworks</a>. Then you need to install the R bindings for GDAL. When you try and install RGDAL from source - you may get this error:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Error: gdal-config not found
The gdal-config script distributed with GDAL could not be found.
</code></pre>
</div>
<p>To let R know where gdal and proj are - try this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>install.packages("rgdal", configure.args = " \
--with-gdal-config=/Library/Frameworks/GDAL.framework/unix/bin/gdal-config \
--with-proj-include=/Library/Frameworks/PROJ.framework/unix/include \
--with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib \
",type="source")
</code></pre>
</div>
<p>R packages install super fast for me, rgdal took about 30 seconds to build and install. I’m beginning to see why people like this package.</p>
<h2 id="plotting-california-state-boundies">Plotting california state boundies</h2>
<p>Download the <a href="http://www.census.gov/geo/www/cob/co2000.html">california shape files</a> from the US census, then load the data and plot it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cali=readOGR("/tmp","co06_d00")
plot(cali)
</code></pre>
</div>
<p>You should get a graph of californian country boundaries:</p>
<p><img src="/images/choropleth/cali.png" />
<cite>Plot of US census geo data</cite></p>
<p>Now to extract data from my twitter data.</p>
<h2 id="generating-neighbourhood-shapefiles">Generating neighbourhood shapefiles</h2>
<p>As I import tweets from the twitter firehose, I create records in my place table that store the geometry data in a postgis <code class="highlighter-rouge">polygon</code> field. This means I can use <code class="highlighter-rouge">pgsql2shp</code> to extract the geometry data and create shape files that can be loaded into arcgis explorer, or imported into R using the readOGR command.</p>
<p>To generate my shape file - I run something like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>pgsql2shp -f london -u postgres tp_dev "select \
name, geom \
from \
places \
where \
kind='neighborhood' and city_id=( \
select id from cities where name='London' \
)"
</code></pre>
</div>
<p>Which gives me 115 rows, all of the neighbourhoods in London that I have crawled so far. Plotting this data in R:</p>
<p><img src="/images/choropleth/london.png" />
<cite>Plot of London twitter neighbourhoods</cite></p>
<p>Twitter uses <a href="http://www.maponics.com/products/gis-data/neighborhood-boundaries/overview/">third party data</a> to geocode your tweets. Sadly, the neighbourhood data isn’t free, so twitter can only share bounding boxes, which is why the map is all boxes.</p>
<p>In the future I might try querying openstreetmap for <a href="http://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative">boundary data</a> of administration areas 9 and 10, but I’m not sure how complete that data is.</p>
<h2 id="getting-tweet-frequency">Getting Tweet frequency</h2>
<p>I re-ran the neighbourhood shape file dump, to include the number of users tweeting from each neighbourhood in london. The query ended up looking something like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>select \
id, geom, name, ( \
select count(distinct user_id) from tweets where place_id=places.id \
) as user_count \
from \
places \
where \
city_id=( \
select id from cities where name='London' \
) AND kind = 'neighborhood' \
order by \
user_count desc;
</code></pre>
</div>
<p>I can then load this data into R, create a blue palette and plot it in 3 lines:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>pal <- brewer.pal(9,"Blues")
plot(lon,col=pal[9 - (lon$USER_COUNT * 9 / max(lon$USER_COUNT))],border="white")
</code></pre>
</div>
<p>The graph however - is kinda disappointing, since the overlapping bounding boxes - combined with some very large neighbourhood areas, makes the graph pretty much useless.</p>
<p><img src="/images/choropleth/london-heatmap.png" />
<cite>Darker blue represents more tweets</cite></p>
<h2 id="two-dimensional-histogram">Two dimensional histogram</h2>
<p>After failing at my first graphing experiment, I tried plotting a two dimensional histogram of tweets. First up - spit out all the lats and longs of tweets from postgres. I haven’t installed the r<->postgres bindings yet, so I have to export to CSV and then import to R.</p>
<p>Enable CSV export from psql.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>\f ','
\a
\pset footer off
\o /tmp/tweets.csv
</code></pre>
</div>
<p>Then query…</p>
<div class="highlighter-rouge"><pre class="highlight"><code>select
st_x(geom) as x, st_y(geom) as y, extract(hour from created_at) as hour
from
tweets
where
geom is not null
and
st_x(geom) < -100
</code></pre>
</div>
<p>This also gives us the hour of the day that the tweet was made, which might come in handy later. I loaded this data into R, then I could plot a scatter graph:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>tweets <- read.csv("/tmp/tweets.csv")
plot(tweets$x, tweets$y)
</code></pre>
</div>
<p>Gives a random looking graph with a concentration of points around the mission.</p>
<p><img src="/images/choropleth/scatter-sfo.png" />
<cite>A scattergraph of 1600 geotagged tweets around San Francisco</cite></p>
<p>Okay - what does this look like as a 2d histogram?</p>
<div class="highlighter-rouge"><pre class="highlight"><code>hist2d(tweets$x,tweets$y)
</code></pre>
</div>
<p><img src="/images/choropleth/heat-sfo.png" />
<cite>A 2d histogram of tweets around San Francisco</cite></p>
<p>Starting to get interesting - but still way too sparse. Let’s trim the area down to only include San Francisco proper, from the Marina District in the north, to the mission in the south.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>city <- subset(tweets, y < 37.808 & y > 37.750 & x < -122.370 & x > -122.457)
</code></pre>
</div>
<p>Then let’s plot it in perspective 3d</p>
<div class="highlighter-rouge"><pre class="highlight"><code>h <- hist2d(city$x, city$y, nbins=25)
persp(
h$x,
h$y,
h$counts,
ticktype="detailed",
theta=30,
phi=30,
expand=0.5,
shade=0.5,
col="cyan",
ltheta=-30
)
</code></pre>
</div>
<p><img src="/images/choropleth/sfo-persp.png" />
<cite>User count in San Francisco rendered in a perspective view</cite></p>
<p>The mission / cbd is in the top right of this graph, so you can see where most people are during the day.</p>
<h2 id="animating-the-graph-over-time">Animating the graph over time</h2>
<p>This is rendering 5 hours of data on one graph, but I was interested to see how the activity looked on an hourly basis, so I wrote a little R script to re-render the graph for each hour of the day, which may be a topic for tomorrows blog post.</p>
<p>nb: Note that I never actually created realistic looking choropleths in this post. ;D</p>
Specialization and Art2010-09-03T00:00:00+00:00http://bnolan.github.com/2010/09/03/art-on-fridays<p>I like <a href="http://elise.com/quotes/a/heinlein_-_specialization_is_for_insects.php">Heinleins quote</a>, so instead of Follow Friday, I decided to try and post Art on Friday. So, an old charcoal drawing that’s been hanging in my office, and 90 seconds of my acoustic guitar.</p>
<p><img src="/images/charcoal-drawing.jpg" />
<cite>Drawn at <a href="http://www.vincents.co.nz/">Vincents</a> life drawing class in 2008</cite></p>
<p>A warning - I made the guitar track up as I went along, so it’s pretty rough eh.</p>
<audio src="/tracks/friday-acoustic.mp3" controls="true" autobuffer="true"> </audio>
<p><cite>Something around Am and C, on my Tanglewood acoustic</cite></p>
<p>The track was recorded by the imac’s internal microphone (I need to get a 9 volt battery so I can use the pick-ups on the acoustic), so sounds a bit like it was recorded in a tin can.</p>
What to drink on Thursdays?2010-09-02T00:00:00+00:00http://bnolan.github.com/2010/09/02/what-to-drink<p>I’ve been making a real effort to avoid coffee in the afternoon. I’ve made the discovery that although the first cup of coffee in the morning is awesome, afternoon coffees are just that ‘cup of stress’ that my vegan friends talk about.</p>
<p>So I made myself a handy guide to what I should drink, hour by hour, on Thurdays.</p>
<p><img src="/images/drinks.jpg" />
<cite>Tea… Wins!</cite></p>
<p>The beer is optional, although I was planning on catching up with <a href="http://chillu.com/">Herr Schommer</a> after work.</p>
Polygons and postgis2010-09-01T00:00:00+00:00http://bnolan.github.com/2010/09/01/polygons-and-postgis<p>I use <a href="http://postgis.refractions.net/">postgis</a> extensively in my geo-local development. If you’re working on a project that involves geo data and you haven’t seriously looked into postgis, you’re doing yourself a disfavour.</p>
<p>Postgres is really quite epic.</p>
<ul>
<li>Postgis</li>
<li><a href="http://rpgsql.sourceforge.net/">R for Postgres</a></li>
<li><a href="http://www.pygresql.org/">Python for Postgres</a></li>
<li>A nice stable API for creating C++ extensions</li>
</ul>
<p>We used the C++ API to make a <a href="http://xapian.org/">xapian</a> plugin for <a href="http://zoomin.co.nz/">zoomin</a>, back in the day before <a href="http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/">tsearch2</a> became integrated into Postgresql.</p>
<h2 id="twitterplaces">Twitterplaces</h2>
<p><a href="http://twitter.com/sminnee/">Sam</a> asked that I add polygons onto my <a href="http://twitterplaces.com/sfo/soma">twitterplaces maps</a>, so that he could see where the boundaries of a neighbourhood are.</p>
<p><img src="/images/soma-tp.png" /></p>
<p><cite>South of Mission on Twitterplaces</cite></p>
<p>The twitter places API gives you the bounding boxes of a place, so I already had the information, it was just a case of how to store the bounding boxes. I decided to do it properly and use <a href="http://github.com/fragility/spatial_adapter">spatial_adapter</a> and a :polygon column in my places table.</p>
<p>This meant I could use a method like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def bounding_box=(params)
points = []
params['coordinates'].first.each do |c|
points.push GeoRuby::SimpleFeatures::Point.from_x_y(c.first, c.last)
end
self.latitude = points.collect(&:y).sum / points.length
self.longitude = points.collect(&:x).sum / points.length
c = params['coordinates'].first.first
points.push GeoRuby::SimpleFeatures::Point.from_x_y(c.first, c.last)
self.geom = GeoRuby::SimpleFeatures::Polygon::from_points([points])
end
</code></pre>
</div>
<p>To convert twitters json representation of a place into a WKRT geometry object in my database. This is cool, because then I can use some coffeescript like this to draw the polygon onto my maps.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if $place.geom.rings
points = []
bounds = new google.maps.LatLngBounds
for point in $place.geom.rings[0].points
coord = new google.maps.LatLng(point.y, point.x)
bounds.extend coord
points.push coord
if bounds.toSpan().lat() == 0.0 or bounds.toSpan().lng() == 0.0
marker = new google.maps.Marker {
map : map
position : bounds.getCenter()
}
else
poly = new google.maps.Polygon {
paths : [points]
map : map
strokeColor : '#ff0000'
strokeOpacity: 0.5
fillOpacity : 0.1
fillColor : '#ff0000'
}
</code></pre>
</div>
<p>That code also checks whether the polygon is degenerate (of zero size), and then plots a marker instead. Twitter always sends polygons, even if the polygon has zero area. Anyway. So this was all nice and took 15 minutes to code up. But it also let me do something super awesome:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def find_child_pois
Place.find(:all, :conditions => ['kind = ? AND st_contains(?, geom)', 'poi', geom])
end
</code></pre>
</div>
<p>I now have a method to find all places contained inside a place. So for example, <a href="http://twitterplaces.com/sfo/soma">South of Mission</a> in San Francisco, has <a href="http://twitterplaces.com/sfo/adobe-systems-san-francisco">Adobe Systems</a> inside of it. You can also find out which places contain this one - for example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>>> Place.find_by_path('sfo/soma').find_parents.collect(&:name)
=> ["San Francisco", "SoMa", "California"]
</code></pre>
</div>
<p>I was pretty excited when all this came together so well. When I’ve got more time, I’d like to fix up the crawler that feeds twitterplaces, since at the moment it just dumps the database and recreates it every hour from a sampling of the twitter firehose.</p>
<p>Ah - for more good hours in the day.</p>
Agile board in html52010-08-30T00:00:00+00:00http://bnolan.github.com/2010/08/30/agile-board<p>I put <a href="/science/velocity/">this together</a> as a quick prototype of what an agile board in HTML5 could look like. I’m not a huge agile fan myself, but since I keep having to use pivotal tracker in projects - I’m always tempted to do a little bit more work on it and get it to a usable state.</p>
Batched save ruby module2010-08-29T00:00:00+00:00http://bnolan.github.com/2010/08/29/batched-save<p>In my work on <a href="http://twitterplaces.com/">twitterplaces</a> - I’ve been commonly importing several thousand tweets at once from the streaming API. As I re-write my import process, I have to re-run this import job, so I’ve been trying to get it as fast as possible. Currently what I do is:</p>
<ul>
<li>Use memcache</li>
<li>Use <a href="http://gist.github.com/555750">batched_save</a></li>
</ul>
<p>Batched save is my module I created to let you <code class="highlighter-rouge">include BatchedSave</code> in a model, and then when you save a new record - it’ll batch up 100 records and use COPY to insert them. It is for postgresql only, and obviously only works in situations like mine where the data isn’t self-referential, but it’s mega fast.</p>
My 10k submission was accepted2010-08-27T00:00:00+00:00http://bnolan.github.com/2010/08/27/10k-submission-accepted<p>Woohoo! Please go <a href="http://10k.aneventapart.com/Entry/394">vote now</a>! It’s a bit lame they’ve got the unsexiest screenshot first, but… yay!</p>
<p><img src="/images/10kgallery.png" />
<cite>My entry in the official 10k gallery</cite></p>
<p>There is a community prize that you win by getting the most votes - so if you like my submission, please spread the word.</p>
All done - in under 10240 bytes2010-08-26T00:00:00+00:00http://bnolan.github.com/2010/08/26/under-10240-bytes<p>Well, it’s done. I just submitted my entry to the <a href="http://10k.aneventapart.com/Entry/394">10k apart</a> competition. It’s been a pretty intense 24 hours, but I’m glad I gave myself some extra time this morning to polish up my entry.</p>
<p><img src="/images/10kapart-preview.png" />
<cite>Planning a trip to Paris, a screenshot from an earlier build</cite></p>
<p>You can <a href="http://10k.aneventapart.com/Entry/394">try it out here</a> - all 9766 bytes of it.</p>
<h2 id="toolchain">Toolchain</h2>
<p>I built the app using:</p>
<ul>
<li><a href="http://jashkenas.github.com/coffee-script/">Coffeescript</a></li>
<li><a href="http://lesscss.org/">Less CSS</a></li>
<li><a href="http://api.jquery.com/">jQuery</a></li>
<li><a href="http://diveintohtml5.org/">HTML5</a></li>
<li><a href="http://pdfcrowd.com/">PDFCrowd</a></li>
<li><a href="http://code.google.com/apis/maps/documentation/staticmaps/">Google Static Maps</a></li>
<li><a href="http://developer.yahoo.com/yui/compressor/">YUI Compressor</a></li>
<li><a href="http://code.google.com/closure/">Google closure</a></li>
</ul>
<p>I have some extra code that is commented out, that does an autocomplete from Wikipedia when you add a new destination to your trip. So for example, you could type ‘Montmar’ and get an autocomplete for Montmartre. It’s nice because [a] I’m not very good at french spelling and [b] you automatically get some description text of the place from Wikipedia.</p>
<p>I also removed a column on the left that let you organise trips to different cities. I probably would’ve been pushing the 10k limit had I kept adding features, but mainly I didn’t have time to build and test the features properly. I enjoyed having time today to add some extra destinations to the trip, change the dates a little, and test the app thoroughly on each of the browsers.</p>
<h2 id="compression">Compression</h2>
<p>I’d like to thank <a href="http://blog.nihilogic.dk/2008/05/compression-using-canvas-and-png.html">Nihilogic</a> for their awesome canvas compression technique. Because I could compress the javascript and css so effectively, I didn’t have to muck around pruning bytes from my css and javascript, which meant I could spend more time polishing the app.</p>
<p>For reference - here’s the breakdown of the sizes of my CSS at different stages of my development pipeline. I used a simple bash script to compile, compress and package everything.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>| | Source | Comp | Opt | PNG |
| Less CSS | 7143 | 7589 | 5961 | 1898 |
| Coffeescript | 11388 | 13742 | 9445 | 3879 |
</code></pre>
</div>
<h2 id="internet-explorer-9">Internet Explorer 9</h2>
<p>It’s been an interesting experience using IE 9. It looks like it’s going to be a good browser. I will still use Safari as my main browser and development tool (even if they did release IE9 for mac), but it looks like I’ll be supporting Safari 5, Chrome, Firefox 3.5 and IE 9 in all my future web apps.</p>
<h2 id="thanks">Thanks</h2>
<p>I’d like to thank @sminnee, @ristari, @nzkoz, @clarketus, @nikz, @agrath, @terrcin, @lancehodges, @youdoham, @nazdrug and @campbell for their support, feedback and bug checking while I put this thing together. I couldn’t have got it done without you all, thanks.</p>
Why not consult?2010-08-25T00:00:00+00:00http://bnolan.github.com/2010/08/25/why-not-consult<p>In the preparation for my leap back into the world of start-ups, I’ve been wondering why I don’t prefer to consult full-time. Consulting has a bunch of pros:</p>
<ol>
<li>Excellent pay</li>
<li>Challenging projects</li>
<li>Turn your brain off at 5pm</li>
</ol>
<p>It struck me today - that the main reason I don’t enjoy doing javascript / rails consulting is because I <em>can’t do it all day long</em>. I can do 4-5 hours of it and stay in flow, but after that it’s a real struggle. Sure I can churn out more code and fix bugs, but I’m not being creative, I’m overexercising the part of my brain that writes code.</p>
<p>And in a small business, that’s the perfect time of day to put away your text editor, open up skype, imovie and mail.app, and start down some of that marketing work.</p>
<p>I think this all comes back to Heinlein.</p>
<quote>Specialization is for Insects</quote>
<p>I want to be able to play the cello, climb a multipitch, fix a car, nurse a child, write a book, rally a crowd, run a marathon, pontificate on a beer, console a friend or any of a million more things.</p>
<p>Small business takes away your safety net, but it lets you try out as every act in the circus.</p>
My 10k apart submission2010-08-24T00:00:00+00:00http://bnolan.github.com/2010/08/24/10k-apart<p>I haven’t had much time to look at the Facebook Places API, but I have been working on my <a href="http://10k.aneventapart.com/">10k apart</a> submission. This competition challenges you to build a functioning web app in only 10 kilobytes of javascript, html and css.</p>
<p>I’ve decided to take an idea I’ve had kicking around (a trip planner for <a href="http://www.weheartplaces.com/">weheartplaces</a>) and see if it can be polished up to fit into the 10k limit.</p>
<p><img src="/images/10kapart-preview.png" />
<cite>Planning a trip to Paris, a screenshot from earlier this morning</cite></p>
<p>Using the command <code class="highlighter-rouge">ls -l *.css *.js *.html | awk '{sum+=$5} END {print sum}'</code> to sum up the bytes, I’m already up to 12k of content for the unoptimized code.</p>
<p>I’m aiming to keep the entire project under 20k in unoptimized form, which will hopefully make the 10k limit achievable. <a href="http://jashkenas.github.com/coffee-script/">Coffeescript</a> and <a href="http://lesscss.org/">less</a> output is quite verbose - and will benefit from <a href="http://code.google.com/closure/">closure</a> and the <a href="http://developer.yahoo.com/yui/compressor/">YUI compressor</a>.</p>
<p>Wish me luck. The deadline is lunchtime tomorrow (NZDT).</p>
A varied gym schedule2010-08-23T00:00:00+00:00http://bnolan.github.com/2010/08/23/a-varied-gym-schedule<p>Like some web developers, I’m a bit ADD, so I find going to the gym a bit of a chore. Don’t get me wrong. I love the exercise, the clear headedness afterwards, the more energy you have in every day, the rest for your mind - a blissful 45 minutes where your brain doesn’t have to analyse pixels on a screen, or type strange patterns into the keyboard.</p>
<p>But spending 45 minutes with no mental stimulation - how dull is that! Especially when you go to the gym at quiet times (I don’t like noisy gyms), there’s not even anyone to talk to.</p>
<p>So my workout which is nice and varied is like so:</p>
<ol>
<li>3km rowing ~12 minutes</li>
<li>Situps, pushups and chins x3 sets ~15 minutes</li>
<li>Treadmill on hilly mode ~15 minutes</li>
</ol>
<p>I find the treadmill the hardest, since it’s just so dull - but I take it as a nice time to be all Zen (after the half an hour of exercise I find my inner Zen-buddha is easier to access). I set the treadmill to ‘randomly changing gradient’, so that your heart rate is constantly moving up and down.</p>
<p>Anyway - that works for me, and I’ve found it an easy workout to do consistently 4 times per week</p>
Analyzing a places page2010-08-20T00:00:00+00:00http://bnolan.github.com/2010/08/20/a-facebook-place<p>Since Facebook places isn’t available outside of the US yet - I’ve only had two options for exploring the places api.</p>
<p>Firstly - my good friend <a href="http://twitter.com/clarketus/">loliver</a> moved to Palo Alto two months ago to work for <a href="http://www.wildfireapp.com/">Wildfire</a>, so I’ve been following his <a href="http://www.facebook.com/pages/Wildfire-HQ/119484554770490">checkins</a>.</p>
<p><img src="/images/facebookplace-page.png" />
<cite>Wildfire HQ Places page</cite></p>
<p>I love the design, it’s nice and simple, has a strong social component (with the facebook wall), had a piece of content in the middle of the page (the map), some stats on the left to get an idea of the popularity of this place - and then a list of people who have checked in there which gives some ‘explorability’ to the places pages.</p>
<p>They don’t have ‘nearby’ or ‘similair’ places, but that might just be because it’s new and not enough of my friends have checked into multiple places.</p>
<h2 id="proxies-and-geolocation-hacks">Proxies and geolocation hacks</h2>
<p>The other way I’ve been trying out the facebook is by proxying my requests through my US servers (which doesn’t seem necessary now that they’ve deployed the places functionality worldwide). The only downside is that when your browser tries to geolocate you, Facebook works out you’re not in the US and gives up.</p>
<p>The next trick for Monday will be to make a greasemonkey script that emulates the W3C geolocation API and let’s you check <a href="http://twitter.com/nzkoz/">friends</a> into <a href="http://www.tuscl.net/s.php?SID=27">Kentucky strip clubs</a>.</p>
Facebook Places2010-08-19T00:00:00+00:00http://bnolan.github.com/2010/08/19/facebook-places<p>I’m really excited about the release of <a href="http://developers.facebook.com/blog/post/403">Facebook Places</a>. It’s not widely available yet but it should be rolled out to American Facebook users over the next 24 hours.</p>
<p><img src="/images/facebookplaces.png" />
<cite>Facebook places on the iPhone</cite></p>
<p>The features:</p>
<ol>
<li>Checking in to places, for where you are at the moment</li>
<li>Tagging friends as being with you at a place</li>
<li>Seeing where your friends are and have been</li>
<li>An extensive API for accessing a users places data</li>
</ol>
<p>It’s number 4 that’s exciting to me. Facebook gives a bunch of examples of ways the API could be used - including:</p>
<p><em>A travel application that gives people the ability to see which of their friends have already been to the place they are visiting.</em></p>
<p>That’s kind of exactly what I’ve been building. Starting next week, I’ll be working on the re-release of <a href="http://www.weheartplaces.com/">weheartplaces</a>, integrated with the Facebook Places api.</p>
We Heart Nokia2010-08-18T00:00:00+00:00http://bnolan.github.com/2010/08/18/we-heart-nokia<p>I’m excited to announce that weheartplaces will be available for Nokia Series 60 phones, as well as the iPhone and Android.</p>
<p>I’ll be using the Nokia Webruntime (WRT 1.0), a browser-based environment that effectively allows you to create web/javascript programs that masquerade as normal symbian applications. I’ve just been testing out the functionality of my <a href="http://www.nokia.com.au/find-products/all-phones/nokia-6220-classic?cid=ncomprod-fw-src-na-con-nokia_6220-google-au-20-nokia6220_117">Nokia 6220</a>, which is a $150 phone with 3G, GPS, driving directions and a 5 megapixel camera.</p>
<p><img src="/images/weheartnokia.jpg" />
<cite>I got the icon going! That’s like 80% of the work right there.</cite></p>
<p>I’m not sure of the release date yet - but when it’s ready it’ll be available at my <a href="http://store.ovi.com/publisher/Nolan+Consulting/">publishers page</a> on the Ovi store.</p>
To Market!2010-08-17T00:00:00+00:00http://bnolan.github.com/2010/08/17/to-market<p>Are you in Wellington and are serious about starting a tech business? I got an email from the <a href="http://www.brightideaschallenge.co.nz/">bright ideas challenge</a> with information about the following presentations, from various technical and business development peoples. I’ll be attending these three…</p>
<h2 id="strategy">Strategy</h2>
<p>Value proposition, business model, pricing model, finances and sustainability - Monday, 30 August 2010.</p>
<h2 id="technical">Technical</h2>
<p>Intellectual property strategy, legal and governance, prototyping and funding options - Monday, 6 September 2010.</p>
<h2 id="go-to-market">Go-to-Market</h2>
<p>Go-to-market strategy, branding, exporting and distribution - Wednesday, 8 September 2010.</p>
<p>All events are held 4.30-6.30pm in the Memorial Theatre, Victoria University of Wellington. That marketing one sounds especially interesting.</p>
Weird bug of the day2010-08-16T00:00:00+00:00http://bnolan.github.com/2010/08/16/weird-bug-of-the-day<p>On Safari, it’s nice to use ellipsis to limit the length of text. However, a warning to young players - <code class="highlighter-rouge">text-overflow: ellipsis</code> doesn’t work with webfonts.</p>
<p><img src="/images/overflow-1.png" />
<cite>Correctly functioning ellipsis styling</cite></p>
<p><img src="/images/overflow-2.png" />
<cite>When using a webfont, Safari gives up, it’s all too hard</cite></p>
Hey what does this button do?2010-08-15T00:00:00+00:00http://bnolan.github.com/2010/08/15/hey-what-does<p>I’ve used photoshop for years and consider myself an intermediate user. You end up using photoshop a lot in webdesign to slice stuff up, or create decent logos - and prior to CSS3 it was my number one gradients tool. However - there’s a bunch of stuff photoshop doesn’t do.</p>
<p>With the advent of resolution-independent interfaces (retina display, next version of os x, landscape and portrait views of a site) and the need to create business cards, flyers, stickers and pixel art for websites - I’ve been learning Adobe Illustrator.</p>
<p>While playing with the <a href="http://www.weheartplaces.com/">weheartplaces</a> logo - I discovered this menu item:</p>
<p><img src="/images/logos/menu.png" />
<cite>Hey - what does this effect do?</cite></p>
<p><img src="/images/logos/cut6.png" />
<cite>Houston. We have a logo.</cite></p>
Logo Design2010-08-14T00:00:00+00:00http://bnolan.github.com/2010/08/14/weheartplaces<p>I’ve been working on logos and design rules for <a href="http://www.weheartplaces.com/">Weheartplaces</a>, creating dozens of Artboards in illustrator. I’m not a designer by trade, so some of my design is a bit stilted, but bear with me and eventually there might be a nice logo and style for the site.</p>
<p><img src="/images/logos/cut1.png" />
<cite>Dax, probably too strong a shadow.</cite></p>
<p><img src="/images/logos/cut2.png" />
<cite>Eurostile, going cold on this one.</cite></p>
<p><img src="/images/logos/cut3.png" />
<cite>Oh, hai Steve.</cite></p>
<p><img src="/images/logos/cut4.png" />
<cite>The Scott Pilgrim special, plus Illustrator gradients.</cite></p>
<p><img src="/images/logos/cut5.png" />
<cite>For legal correspondence only.</cite></p>
Push state and Node.js2010-08-13T00:00:00+00:00http://bnolan.github.com/2010/08/13/pushstate-and-nodejs<p>I’ve talked with <a href="http://twitter.com/sminnee">Sam Minnee</a> (CTO at <a href="http://silverstripe.com/">Silverstripe</a>) a bunch of times about running jquery on the server side. It was only a hunch, I never knew exactly what the benefits would be - but with the availability of <a href="https://developer.mozilla.org/en/DOM/Manipulating_the_browser_history">pushState</a> on Firefox, Safari and Chrome - it suddenly makes a lot of sense.</p>
<p>What pushState and jquery on the server lets you do - is to have a pure javascript application that doesnt require page reloads to go to different urls. For example:</p>
<ol>
<li>http://yourapp.com/</li>
<li>http://yourapp.com/search</li>
<li>http://yourapp.com/search/myterms</li>
<li>http://yourapp.com/tweet/12345</li>
</ol>
<p>Can all be generated client side by your jquery and coffeescript code. This is an awesome way to write an app, you get a good responsive app and it’s easy to persist state between pages (eg your google map won’t spring back to the default location everytime you click ‘back’).</p>
<p>The number one downfall of javascript apps, is that there are now pages in your app that can’t be linked to - nor can they be crawled by google, or scraped by any other site that uses the http protocol. Building exciting new apps that ‘break the internet’ isn’t cool.</p>
<p>But if you can run your entire app on the server side - then if someone goes to <code class="highlighter-rouge">http://yourapp.com/search/myterms</code> - node.js can load your page on the server, recognize the url, run the correct javascript code to update the state as the browser would - then send it down the wire. So that non-javascript clients can access the same content that a desktop browser can.</p>
<p>It’s a whole new way to do accessibility.</p>
Facebook places2010-08-12T00:00:00+00:00http://bnolan.github.com/2010/08/12/facebook-places<p>This <a href="http://gigaom.com/2010/08/10/facebook-places-will-be-about-more-than-just-a-check-in/">analysis by Gigaom</a> is interesting to me. I’m really keen to see what Facebook does with places. What they’ve done so far has been underwhelming technically, but has seen impressive takeup.</p>
<p>For example - try searching for <a href="http://apps.facebook.com/weheartplaces/places/935148524-hashigo_zake">Hashigo Zake</a> (a Wellington beer bar popular with early adopters / geeks) on various services:</p>
<ul>
<li><a href="http://www.facebook.com/pages/Wellington-New-Zealand/Hashigo-Zake/144268315585">Facebook page</a> - 413 people, 100s of posts</li>
<li><a href="http://foursquare.com/venue/306746">Foursquare</a> - 65 people, 3 tips</li>
<li><a href="http://gowalla.com/spots/571126">Gowalla</a> - 10 people, 4 comments</li>
<li><a href="http://maps.google.com/maps/place?cid=11598129844502475571&q=hashigo+zake&hl=en&cd=1&ei=BSBiTLDxDYOEvgPfzKSqDg&sig2=RxTDHaj8pywo70MGfMzcSg&dtab=2&pcsi=11598129844502475571,1&geocode=FX7wif0douxqCg&sll=-41.291693,174.779499&sspn=0.004184,0.003358&ie=UTF8&ll=-41.289126,174.777267&spn=0,0&z=17&iwloc=A">Google places</a> - 5 maps - 2 reviews</li>
<li><a href="http://search.twitter.com/search?q=hashigo+zake">Twitter</a> - 4 tweets in the past week</li>
</ul>
<p>Facebook has such massive uptake, that whatever it does has an impact. The understanding, technology and devices behind ‘location based’ services are so widespread that it’s really a case of Facebook just providing the tools and they’ll probably dominate existing location based services.</p>
<p>Also - note that all of the sites have reasonably terse (or even descriptive) URLs. All except Google. They really don’t get clean URLs.</p>
Make LESS less painful2010-08-11T00:00:00+00:00http://bnolan.github.com/2010/08/11/less-painful<p>I’ve been using <a href="http://lesscss.org/">LESS</a> to make writing css for a <a href="http://jqtouch.com/">jqtouch</a> app less painful. However - the less compiler is a bit awkward to use by default - if you run:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lessc -g -w *.less
</code></pre>
</div>
<p>It recompiles your scripts everytime you make a change, which is perfect, and it notifies you by growl if you have a syntax error - wonderful.</p>
<p>However - every time you get a syntax error, it also blocks the terminal that you launched less from, and you have to go to that terminal and ‘Press [return] to continue…’. It’s pretty annoying. The solution is to load up <code class="highlighter-rouge">/Library/Ruby/Gems/1.8/gems/less-1.2.21/</code> in your text editor, search for command.rb, and change the <code class="highlighter-rouge">run!</code> method, to disable the <code class="highlighter-rouge">$stdin.gets</code> and replace it as so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code># File has changed
if (not File.exists?( @destination )) or File.stat( @source ).mtime > File.stat( @destination ).mtime
print Time.now.strftime("%H:%M:%S -- ") if @options[:timestamps]
print "Change detected... "
if parse
# ...
else
`touch #{@destination}`
end
end
</code></pre>
</div>
<p>I also set the growl.priority to 1 in <code class="highlighter-rouge">err(...)</code> to make my notifications show up in red.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>growl.priority = 1
</code></pre>
</div>
<p>You then get growl notifications on error, but simply correct the error and hit save again to recompile. No more hunting to find the right terminal.</p>
CoffeeScript 0.92010-08-09T00:00:00+00:00http://bnolan.github.com/2010/08/09/coffeescript-0.9<p><a href="http://jashkenas.github.com/coffee-script/">CoffeeScript 0.9</a> was released over the weekend. CoffeeScript is a compiler that turns .coffee files into .js files. CoffeeScript is a significant whitespace, rubyish language that makes large (and small) javascript projects much more manageable and fun.</p>
<p>I’ve been using CoffeeScript since 0.5, first with a multiplayer javascript game experiment, then more recently as part of my consulting with <a href="http://lplabs.com/">Lonely Planet Labs</a>. It’s been interesting working with multiple developers on CoffeeScript - and seeing how it works in a production environment. I can heartily recommend it to teams that are looking to ease their javascript development.</p>
<p>CS 0.9 is a big change from 0.7.2, read <a href="http://github.com/jashkenas/coffee-script/issues/closed#issue/541">this ticket</a> for some of the background on the change. My own reaction to the change is interesting - I was against the change at first, and only changed my mind after 0.9 was released. I was impressed by how <a href="http://github.com/jashkenas/">Jeremy</a> got opininions and analysis from all the existing users, then quickly made such a fundamental change to the language, let people try it out in a branch - and finally decided to merge it back into stable.</p>
<p>All this - despite the fact that upgrading most coffeescript apps to 0.9 is non-trivial. Changing the symbol used for assignment will probably break most existing code. The assignment change was required to bring in yamlish object assignment which is awesome. The new release also has a host of other, non contentious improvements, around superclasses.</p>
<p>I think it’s a good reminder, that no matter how substantial a change - if you manage is right, and the benefits outweigh the costs, your community will adapt and embrace the new version.</p>
Safari Extension2010-08-08T00:00:00+00:00http://bnolan.github.com/2010/08/08/safari-extension<p>I’ve been working through the extension authoring documentation for each of the browsers. I was excited when I got my little icon showing up in the Safari toolbar for the first time:</p>
<p><img src="/images/safari-preview.png" />
<cite>Weheartplaces toolbar button</cite></p>
Design features2010-08-05T00:00:00+00:00http://bnolan.github.com/2010/08/05/design<p>While redesigning this blog (I love being able to edit css and template as easily as blog posts, <a href="http://pages.github.com/">go github pages</a>!), I found myself turning this:</p>
<p><img src="/images/design-staccato.png" />
<cite>Ordered chronologically</cite></p>
<p>Into this:</p>
<p><img src="/images/design-wave.png" />
<cite>Ordered by aesthetic</cite></p>
<p>I find the wave a lot more pleasing to the eye, and I still kept the primary order mostly intact. This is one of my design tenets - lay out the page by weight and balance, then work out what text you can fit into each of the boxes.</p>
<p>It’s almost the opposite of how <a href="http://37signals.com/papers/introtopatterns//index">the signals</a> do design, but I love it.</p>
Brewdog Punk IPA2010-08-04T00:00:00+00:00http://bnolan.github.com/2010/08/04/brewdog<p>I just discovered the awe inspiring <a href="http://apps.facebook.com/weheartplaces/cities/1004679663-wellington/places/935148610-flagstaff_cellars">flagstaff cellars</a>, which is a tucked away little beershop in an unassuming corner of Melbourne.</p>
<p>The thing about this place though, is that is a stocker of Brewdog, Rogue, Epic and a number of good Australian beers - such as Matilda Alpha, Little Creatures and Moo Brew.</p>
<p>Anyway - they had this beer on special - <a href="http://www.brewdog.com/punk_ipa.php">Punk IPA</a> by <a href="http://www.brewdog.com/">Brewdog</a>, so I bought two and came home and tried them.</p>
<p><img src="/images/brewdogpunk.jpg" />
<cite>A very good beer.</cite></p>
<p>The internet website <a href="http://www.ratebeer.com/beer/brewdog-punk-ipa/72423/">rate beer</a> said good things about the Punk IPA, although the haters at <a href="http://beeradvocate.com/beer/profile/16315/39435">beer advocate</a> weren’t so enamoured. The guy who drunk a Punk IPA that was 12 months past it’s best date - in particular - has a bad time with the beer.</p>
<h1 id="my-review-of-punk-ipa">My review of Punk IPA</h1>
<p>This beer is awesome. It cost $5. I will drink it again. Probably in a few minutes when I get the other one out of the fridge. <b>5 stars.</b></p>
Cities in the world..2010-07-29T00:00:00+00:00http://bnolan.github.com/2010/07/29/number-of-cities-in-the-world<p>When you’re building an application for travellers, you come up against the problem of how do you get enough data in the system so that it’s useful for 455 cities with a population of more than one million people, 1054 with more than 500,000 and 2851 with more than 150,000 people.</p>
<p>Or to look at the problem a different way - the most visited cities in the world…</p>
<h1 id="most-visited-cities">Most visited cities</h1>
<p>This <a href="http://en.wikipedia.org/wiki/Tourism#Most_visited_cities">table</a> at Wikipedia shows the top <b>10 most visited</b> cities to be <i>Paris, London, Bangkok, Singapore, Kuala Lumpur, New York City, Dubai, Istanbul, Hong Kong, Shanghai</i>.</p>
<h1 id="crawling-data-for-these-cities">Crawling data for these cities</h1>
<p>I’m not sure what the take away from this data is. Seeding 2851 cities with data seems very doable in a week of crawling, seeding the top 10 visited cities with deep data is also achievable - it’d be interesting to compare the list of top 2851 cities with the hometowns of the users of <a href="http://www.weheartplaces.com/">weheartplaces</a>.</p>
<h1 id="the-real-take-away">The real take away</h1>
<p>Is that it’s better to encourage users to add content to ‘empty’ cities, than to expect to have every city to have a persuasive dataset when you launch your travel site.</p>
Haversine in Coffeescript2010-07-28T00:00:00+00:00http://bnolan.github.com/2010/07/28/haversine-in-coffeescript<p>The <a href="http://en.wikipedia.org/wiki/Haversine_formula">Haversine formula</a> is a equation that can be used to calculate the straight line distance between two coordinates on a sphere. It is commonly used by geohackers to work out how far in <em>kilometres</em> it is between two latitude/latitude coordinates.</p>
<p>Remember that lat/longs are actually spherical coordinates expressed in degrees.</p>
<p>This <a href="http://kapelica.hr/kw/latlong.html">page</a> at kapelica.hr contained a great implementation of the haversine formula in javascript, I tidied it up and ported it to coffeescript to measure distances to the nearest places in my iphone app.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Radians: (degrees) ->
degrees / 57.2957795
Haversine: (lat1, lon1, lat2, lon2) ->
R = 6371 # km
dLat: Radians(lat2-lat1)
dLon: Radians(lon2-lon1)
a: Math.sin(dLat/2) * Math.sin(dLat/2) + Math.cos(Radians(lat1)) * Math.cos(Radians(lat2)) * Math.sin(dLon/2) * Math.sin(dLon/2)
c: 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a))
R * c
</code></pre>
</div>
<p>Note that the degrees->radians conversion isn’t exact, but the results should be within the margin of error for most GPS positioning systems.</p>
Displaying only the first paragraph2010-07-28T00:00:00+00:00http://bnolan.github.com/2010/07/28/display-only-the-first-paragraph<p>In redesigning my blog and publishing it on github pages, I wanted a simple way to show only the first paragraph of each post on the front page. The way I did it isn’t something I’m proud of - but it does show the power of css.</p>
<p>The following rule hides everything apart from the first child of a div.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>div > * + *{
display: none;
}
</code></pre>
</div>
<p>It works on safari, firefox and will work on ie9.</p>
Offline manifests2010-07-23T00:00:00+00:00http://bnolan.github.com/2010/07/23/offline-manifests<p>I’ve been working on an html5 app for Lonely Planet, and part of the task is to make the app work offline on the iPhone. Getting it to work isn’t too hard - you need to create a <a href="http://diveintohtml5.org/offline.html">cache.manifest</a> file and make sure it is served with the correct mime type.</p>
<h2 id="the-problem-is-online">The problem is online</h2>
<p>When you have the phone offline and load the app, it loads in 2 seconds. When you turn the phone online again, it takes about 30 seconds to load all the javascript and so on.</p>
<h2 id="its-the-caching-rules">It’s the caching rules</h2>
<p>We have been deploying to google app engine, and by default - app engine sets all files to expire after 10 minutes. So even though the iphone has a local copy of the files, every 10 minutes, it’ll expire the local cache and redownload everything that the manifest refers too.</p>
<h2 id="a-checklist-for-fast-online-iphone-apps">A checklist for fast online iphone apps</h2>
<ol>
<li>Use a manifest</li>
<li>Ensure the manifest is served with the mimetype text/cache-manifest</li>
<li>Ensure the manifest has no 404 errors in it</li>
<li>Ensure the manifest and all the content in it has an expiry date in the distant future</li>
<li>White-list your network resources explicitly in the manifest</li>
<li>Use jqt.offline.js to help debug the browser behaviour</li>
<li>Tail your webserver logs and keep an eagle eye for unnecessary requests</li>
</ol>
<p>All going well, your html5 app will load instantly, and still be able to access network resources.</p>
Coffeescript2010-07-05T00:00:00+00:00http://bnolan.github.com/2010/07/05/coffeescript<p>I’ve been doing a lot of coffeescript lately and can recommend it wholeheartedly.</p>
<p>That is all.</p>
Skating in Melbourne2010-06-20T00:00:00+00:00http://bnolan.github.com/2010/06/20/skating-melbourne<p>I just went on a big skate of melbourne CBD.</p>
<p>I started at home (nähe Southern Cross Station), then skated out to Prudence in West Melbourne. Prudence is a highly recommended bar. The beer is expensive, but the cocktails apparently are affordable. From Souther Cross to Prudence is mostly uphill, which meant that after a few beers, a few conversations and a good laugh - it was all downhill to Section 8.</p>
<p>Section 8 is a very cool bar. During the day. I went there in the evening and wasn’t so impressed. The skate was good though - it’s mostly downhill from Prudence along Victoria Street - then turn right, drop down into town (it’s a pretty steep skate through traffic down to little collins street) - and a hard left into Section 8.</p>
<p>S8 only had one beer I wanted - and it was in what australians call a “tallie” (800ml beer). I took a few drinks - decided S8 wasn’t for me, then stashed the beer in my back pocket and headed through town. There were people everywhere (and you can’t drink in public anywhere in Australia) so it took 5/10 minutes before I got out of the city (skating past the Melbourne Cricket Ground by now) and could drink the rest of my beer. It was a lovely long skate past the MCG - down a shallow smooth slope on a path surrounded in trees and grass. Had some good tunes on - nice times.</p>
<p>Then I bombed from the MCG down to Richmond - eyeballed out the gig I was going to go see at the corner bar, decided it wasn’t worth $50 to go see - then caught a tram back to Southern Cross and mellowed along Spencer street to home.</p>
<p>Melbourne. Good for skating.</p>
Merging identities2010-05-24T00:00:00+00:00http://bnolan.github.com/2010/05/24/merging-identities<p>I’ve set this blog up on Github pages using Jekyll. My old posts may find their way here….</p>