I wanted to call this pitch "Use Frames, It's easy!", but somebody beat me to it, so having fun with frames it is.

Every now and then, a new processor comes on the market and we need an even harder problem to solve (in an even more stupid manner). This time I've chosen to try to create a mosaic for a motion picture. Think mosaics for every frame of a movie (not a movie of a mosaic in a mosaic). Add to that one rule: no cheating. Most mosaic programs cheat by changing the brightness or color of the sub images. To compensate for not doing any of the cheating, we will need a large collection of images to pick from and multiple computers to do the picking. The problem is simple: for a given frame select the best sub-frames to resemble it.

I've written a client and server which will do HSB full image matching using an absolute distance nearest neighbor algorithm. If you don't know what that means, it's the first algorithm you would come up with yourself. This means that if we get enough laptops together, we should at least be able to get the best frame out of a large collection and do it quickly enough to be done before Christmas.

So, I would like to invite everybody to join in an effort to create a video together. I've got the following sessions lined up:

1. Get the code running, which includes:
   1. Install gstreamer and gstreamer-plugins
   2. Compile and run the test
   3. Get to grips with the implementation
   4. Size up the problem
   5. Decide on what to do:
      1. Make successive subframes in the generated mosaic actually successive frames of the original. (my own favorite)
      2. Optimize current implementation and go nuts for a while.
      3. Get the cloud to do the work.
      4. Bring in a new technology (like Hadoop clustering or 0MQ).
      5. Refactor and clean up the code to handle correct aspect ratios and other resolutions
      6. Start a VM and view the results at new years eve.
2. Go with the decision made in the first session.
3. Do the final run of the software an create our "best result".

At these sessions I will make sure we have a switch to connect all the laptops, some videos to use as sub-frame and a working version on my laptop.

Try it at home (on Linux, but Mac should also work):
Install gstreamer and gstreamer plugins good (use brew for the Mac)
Install ffmpeg as a command, to collapse multiple frames back to a movie
$ git clone git://github.com/bneijt/videosaic.git
$ cd videosaic
$ ./test.sh

This should create /tmp/image_?????.png files which contain a mosaic and (if you have ffmpeg installed) create a file called /tmp/image.ogg which contains a videosaic of vid/onesecond.ogg in itself. If you can get the movie to work, you are more then ready for the first session and you will also have seen that there is a long way to go!

This is the first update since we started development on the LogStore. Despite the bumpy ride, we have quite a lot accomplished.

To summarize, the LogStore is a storage solution for logging statements. It’s based on CouchDB (document oriented database) and Elastic Search (schemaless search engine). The view on the log statements stored is built on CouchApp, a tool on CouchDB for creating HTML pages that are served directly from CouchDB instead of a separate application server.

Read the rest of this post »

This week we finally got some real stuff done on the Xebia Blog recommendation center.

Since our main goal for this app is to actually get something working that could be integrated in the Xebia Blog, we decided to go with reasonably familiar technologies (Java, Maven 2, guice 3.0 for dependency injection, AOP, Jersey for REST, JPA, Jetty) and focus on getting end-to-end functionality first:

- Add blogs to some kind of store (via REST)

- Get recommendations back from this store (via REST)

For now, the recommendations are simply a random pick from the set that is stored, but at least we have something working. I must say that after 2 app incubator projects that mainly consisted of people struggling with a bunch of cool but relatively unfamiliar technologies, being able to really get something working within a couple of hours was quite a positive experience (confession: it really helped that Mischa already had a guice configuration from a different project that we could copy into this app). So even though my hands are itching to convert the lot to Scala, I will keep my cool for now and stick with good ol' clunky Java.

What we will focus on next:

- Pulling mechanism for the actual blog

- Getting more input for the recommendation (language processing, google pagerank, ...)

- Integration with WordPress

- Creating a smarter recommendation algorithm

If you want to check it out, the project is on github (https://github.com/xebia/app-incubator-blogrecommender)

With our Web Sockets incubator we had a rough start on Atmosphere grinding through the samples. There is no intention to bash Atmosphere, it looks like an interesting approach. However, it should be a lesson to all aspiring framework builders out there: getting your samples in top shape matters a lot. This week we tried out socket.io and I wasn't too hopeful as I had planned all kinds of things throughout the evening. The result of a 40 minute pairing session with Sonny Gill was pretty good and I'll share the basic highlights here in a very short post.

Last night me, Iwein, Frank, Arjan, and Freek spent some time playing with one of the basic Web Sockets samples of the Atmosphere framework. I had read a few things about Atmosphere before, especially about its ability to provide Web Sockets support while still gracefully degrading back to Comet/long polling/etc. for browsers that don't support Web Sockets yet. We decided to start with the simple sample app called "Atmosphere JQuery PlugIn PubSub with support for WebSocket", which looked like a promising beginning.

Read the rest of this post »

The purpose of the Xebia App Incubator has always been to stimulate innovation, creativity, learning about new technology, and working together with colleagues on cool mini-projects. We started out as a little idea within XSD (Xebia Software Development) but it turns out that people all over Xebia want to join in, so starting January 2011 the App Incubator will be on equal footing with other, more traditional ways to share knowledge with our colleagues (i.e. presentations).

This does make it a little harder to keep track of all the running projects, new project pitches, and results of past projects. Projects can start whenever an initiator finds enough people to form a team and voting will also be done online. I expect that things would soon become very chaotic without some kind of application to keep track of everything. This brings me to a new project proposal me and Friso came up with:

What will we build ?

A public facing webapp for pitching, voting, planning and keeping track of AI projects. It will be a public website viewable to the whole world but editable only by Xebia employees (through Crowd login). It will of course also use some cool new technologies to keep it interesting.

Proposed Features

• Overview of all current projects, recently finished projects, accepted future projects (see UI design below)
• Click-through to project details: 
  • Blog entries
  • Github project
  • Threaded discussion (with email notification and reply-by-email support)
• Overview of current project pitches
  • Logged in users can sign up as core team members or contributors
  • Public users can like or dislike a proposed project
• Login through Crowd
• The ability to create, update, and schedule projects (for logged in users)
• Associate a user with an avatar image through Crowd/Confluence, Twitter, Facebook, Gravatar

I'm sure we can come up with a lots more features as soon as we start actually using this application but lets stick with the above list for version 1.0.

What's the technology stack ?

We will implement this using the already familiar "REST API with HTML5 frontend" approach formerly known as "NoJSP" with MongoDB as the storage engine.

• Server options (choice still to be made): 
  • Node.JS (Javascript. Expert: Friso)
  • Scalatra (Scala. Expert: Age)
  • Spring REST (Java)
• Client-side Dynamic HTML rendering: Mustache (http://mustache.github.com/)
• Storage: MongoDB

What will it look like ?

Below is a little design sketch I did for the front page:

I was inspired by a post on the 37 Signals blog about one of their internal apps. It has the idea of using avatars to show which people are part of a project. Friso came up with the idea of a calendar-like timeline to show an overview of projects.

Who wants to join in ?

Last night we had another "present and pitch" round and it was bigger and better than ever before. The teams from the previous App Incubator round presented their results (with some slick presentations, one of which has already been posted here) and then people pitched their new ideas, all of which have also been posted here in the last few days. All in all there was a lot of creative energy in the room.

There was a surprising number of mathematics-based proposals this time and we even had the first stirrings of a "my mathematics is better than yours" flamewar. It was also great to see more of our colleagues than ever before be present at the pitching of new projects.

So here are the results of the jury:

The above picture shows the seven projects as previously posted here (in my crappy handwriting on one of our infamous see-though whiteboards), followed by the initials of everyone in the room as they pledged their support for one project or another. As it is standing now, the "Profiled News on your Android Phone" proposal unfortunately didn't make it. Two other projects are a bit on the edge with two votes each, but they will probably be fine since about 5 team members were not present last night to cast their votes. The rest made it through the votes with flying colors.

So now that the campaign and the elections are over, it is time to get back to coding. We'll start work on these projects at the first session, which is in two weeks on Tuesday October 26th at the usual location (Xebia offices at Laapersveldt, Hilversum, the Netherlands). See you there !

Yesterday we did our final presentation on the noJSP incubator project. Watch the cool prezi presentation here!
We had a lot of fun, experienced with a new technology and learned a lot.

You might not have realized it yet but we are now squarely in the age of the realtime web. Twitter is migrating towards "streaming" interfaces, Google Docs allows you to interact with coauthors in realtime, Google search has gone "instant", and just about anyone wants to show you realtime analytical data.

But how does all this data get to your browser ?

First we hacked browsers using Ajax polling, then we hacked server side connections and called it Comet, and now, finally, we have a properly designed, non-hacked solution called Web Sockets (see Wikipedia, W3C).

Support for all of the above has slowly made its way to all the browsers and web application servers so how do we go about writing a realtime web application that gracefully degrades from Web Sockets, through Comet, all the way to simple polling without writing a lot of code ? The answer, or at least the Java answer, is called Atmosphere (FYI, the Javascript answer is called Socket.IO, built on Node.js).

So what's the project ?

I'm glad you asked. In order to gain some experience with writing realtime applications, using the best combination of client-side and server-side technologies, we need to write ourselves some Atmosphere code. In order to keep it simple, I propose the following simple application: Counters"R"Us

• the application keeps track of 10 (or pick any number) counters
• these counters are updated at unpredictable intervals by a background job (a simple thread with a semi-random delay)
• each counter is displayed on a unique page
• when your browser has such a page open, it will receive realtime updates for that counter only (i.e. not all updates should go to all browsers)
• use Atmosphere, REST, JSON on the server side
• use the Atmosphere JQuery plugin for the browser side
• we test with loads of concurrent browsers, using the cloud if we have to
• bonus feature: make the counter display the number of active browsers connected to its page

If this is way easier than it sounds, we can try the same thing but then with Socket.IO in Javascript.

Let's get realtime !

Why?

One off the missions in Xebia I heard about is how to become thought leader in Mobile. This simple sentence trigger me, about 4 years ago I had an business Idea. The idea was to create a digital printed newspaper which will be personalized by sending sms messages to a service which could make an profile and match articles which you will also like. Last I saw an initiative which does something which looks a little bit like this, it is named Flipboard and is becoming very popular.
If we can also make such an popular application we will make fame in the Mobile development scenes. 

What
The Idea is to create an interface for Android (and maybe later for other platforms) where you will get:
1. A list of articles ordered by the fact how well your profile thinks you like it (after about 10 - 20 click the profile will work pretty well).
2. A button to add rss feeds which will trigger a process to add RSS feeds and tell where the text of the articles is places on those pages to be able to parse the contents., this button is optional since everyone will see content from all feeds (only the top which they will likely like).

When we create this application and are able to become as popular as Flipboard is becoming now our names will be in the mobile communities!
Why didn't I create this application by myself? I justs do not have enough time.

What do we need for this project?
1. Someone who will create the Android Frontend
2. Someone who will create the service backend
3. Someone to implement the algorithm (I will do some at least)!

What I did before
I once created a demo application based on profiling research. The demo application was able to serve news which got personalised by giving articles an score (-2 --> 2). The personalization will be automatically adjusted and will work based on the following:
1. There will be made a group of groups of interesting words for each article.
2. The profiles will consist of a profile of there groups with scores based on the ratings.

The groups of interesting words will be made by an algorithm which I already created once and tested once. (see: http://onlinelibrary.wiley.com/doi/10.1002/meet.1450430154/abstract)
Basically it works with the following steps:
1. Remove stop words
2. Stemming (each word in the shortest form)
3. Make a non directed graph of biagrams (combine all words off Article A with all words of Article B).
4. Weight all edges with the EAMI-SP function which can e found in the research document).
5. Create as much groups of words by removing edges with low scores.
6. Result: The most imported words grouped by words which mean something to each other.

 

Kind regards,

Mark Bakker
http://www.twitter.com/markbakkernl