Carpathian Cloud

Blogspot #fail acknowledge

2010-07-02T11:49:00.000+01:00

I'll keep this short. I'm tired of workarounds to post what I want the way I want in this blog system. So while this is going to be kept alive on Blogger for reference purposes, I'm going to start using Posterous to publish my random thoughts, academic/professional facts and/or opinions. Thank you for having read along. If you have been enjoying the content, please head on to http://edgargoncalves.posterous.com, and update your feeds to that blog's.

How I live with cold starts on GAE's JVM

2010-06-18T02:40:00.002+01:00

It has come to my attention the (obvious and logical) fact that loading the JVM is expensive, time-wise. And Google's Application Engine (GAE) makes no exception, so whenever one servlet is "hit" long after any previous server interaction, a JVM is started. In practice, this means that we're looking at a "loading..." spinning wheel with a duration of about 10s, +-5s. That's kind of a bother, but that's not the worse. If we're talking about a single-page modeled web app, the user is bound to leave it opened for some time (minutes, not to say hours, before he is able to gather all the info to fill in that devilish form!). And after that time, when he is eager to finish up his task (that has already took him enough time outside the browser window), guess what? That's right, wait another 10s for the server's JVM to wake up. Off course this is a non-issue for heavy-traffic web apps, but this platform is great for experiments and small business products, and it's this niche I'm worried with (perhaps because I'm also targeting it :) )

Today's scenario is yet a bit worse. The time it takes for a GAE JVM to get drowsy is often shorter than 5 minutes (here are some conclusions about timings, as well as other suggestions to alleviate the problem - but insufficient, still). I could only imagine Google is having a hard time escalating their servers to everyone, but they should get a solution to this. Otherwise people will start hacking this problem subversively. In fact, they're already doing it: exploits are there to make a repeated task that keeps the JVM in question alive - a ping, let's call it that! But as soon as this hack becomes mainstream and widespread, Google will hog down with requests. So I'm sort of against it. But since there is no viable solution so far, I started reconsidering things.

What I ended up was a compromise. For applications like the ones I'm developing there will be only 3 or 4 users, and their request will be very scarce and scattered. So I will not use a frequent and constant ping bot. Instead, I made a client-side that does that pinging, but only while the application is opened. So yes, the user has to wait some seconds to fire up the application for the first time. But afterwards, a background (i.e., asynchronous) periodic request keeps the JVM ready to listen for the application's demands.

Here's a possible Javascript code to make this happen. The trick here is to create a script dynamically (i.e., appending it to the body), and setting its source to any servlet's served URL (in this case, a ping one).

International Lisp Conference 2010

2010-06-07T09:50:00.005+01:00

I was asked to spread the word about the upcoming International Lisp Conference 2010 (ILC-2010). So here you have the Call for Arms Papers (from the original attachment):

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   International Lisp Conference 2010

   October 19-21, 2010

   John Ascuaga's Nugget (Casino)

   Reno/Sparks, Nevada, USA (near Lake Tahoe)



   Collocated with SPLASH 2010 (OOPSLA & DLS & more)

   see also http://splashcon.org as well as

   http://www.dynamic-languages-symposium.org/dls-10/



   In association with ACM SIGPLAN (PENDING)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Association of Lisp Users is pleased to announce that the 2010

International Lisp Conference will be held in Reno, Nevada, in

collocation with SPLASH 2010. The scope includes all areas related to

the Lisp family of programming languages.

Accepted papers will be published in the ACM Digital Library (PENDING).

Extended Abstracts and Papers must be written in English and submitted

electronically at http://www.easychair.org/conferences?conf=ilc2010 in

PDF or WORD format. If an Extended Abstract is submitted, it must be

between 2 and 4 pages, with full paper to follow before final deadline.

Final submissions must not exceed 15 pages and need to use the ACM

format, for which templates which can be found at:

   http://www.acm.org/sigs/pubs/proceed/template.html.

Important Dates:

~~~~~~~~~~~~~~~~

* Deadline for Abstract Submission: August 1, 2010

* Deadline for Paper Submission: September 6, 2010

* Author notification: September 20, 2010

* Final paper due (in electronic form): October 5, 2010

* Conference: October 19-21, 2010

Scope:

~~~~~~

Lisp is one of the greatest ideas from computer science and a major

influence for almost all programming languages and for all

sufficiently complex software applications.

The International Lisp Conference is a forum for the discussion of

Lisp and, in particular, the design, implementation and application of

any of the Lisp dialects. We encourage everyone interested in Lisp to

participate.

We invite high quality submissions in all areas involving Lisp

dialects and any other languages in the Lisp family, including, but

not limited to, ACL2, AutoLisp, Clojure, Common Lisp, ECMAScript,

Dylan, Emacs Lisp, ISLISP, Racket, Scheme, etc.

Topics may include any and all combinations of Lisp and:

* Language design and implementation

* Language critique

* Language integration, inter-operation and deployment

* Applications (especially commercial)

* 'Pearls' (of wisdom)

* Experience reports and case studies

* Reflection, meta-object protocols, meta-programming

* Domain-specific languages

* Programming paradigms and environments

* Parallel and distributed computing

* Software evolution

* Theorem proving

* Scientific computing

* Data mining

* Semantic web

We also encourage submissions about known ideas as long as they are

presented in a new setting and/or in a highly elegant way.

Authors concerned about the appropriateness of a topic may communicate

by electronic mail with the program chair prior to submission.

Each paper should explain its contributions in both general and

technical terms, identifying what has been accomplished, explaining

why it is significant, and comparing it with previous work. Authors

should strive to make their papers understandable to a broad audience.

Each paper will be judged according to its significance, novelty,

correctness, clarity, and elegance.

The official language of the conference is English. Some further

information is available at the conference web site, with more details

added later. See: http://www.international-lisp-conference.org

Technical Program:

~~~~~~~~~~~~~~~~~~

Original submissions in all areas related to the conference themes are

invited for the following categories.

* Papers: Technical papers of up to 15 pages that describe original

   results or explain known ideas in new and elegant ways, or extended

   abstracts of 4 pages soon followed by the corresponding full paper.

* Demonstrations: Abstracts of up to 4 pages for demonstrations of

   tools, libraries, and applications.

* Tutorials: Abstracts of up to 4 pages for in-depth presentations

   about topics of special interest for at least 90 minutes and up to

   180 minutes.

* Workshops: Abstracts of up to 4 pages for groups of people who

   intend to work on a focused topic for half a day.

* Panel discussions: Abstracts of up to 4 pages for discussions about

   current themes. Panel discussion proposals must mention panel

   members who are willing to partake in a discussion.

* Lightning talks: Abstracts of up to one page for talks to last for

   no more than 5 minutes.

Depending on the technical content, each submitted paper will be

classified by the program committee as either a technical paper or as

an experience paper; and authors will be informed about this

classification. Note that all interesting submissions are considered

valuable contributions to the success of the ILC series of

conferences. As in past ILC's since 2007, accepted papers in both

categories will be presented at the conference, included in the

proceedings, and submitted to the ACM digital library.

Organizing Committee:

~~~~~~~~~~~~~~~~~~~~~

* General Chair:

   JonL White - The Ginger IceCream Factory of Palo Alto, ALU

* Program Chair:

   Antonio Leitao - Instituto Superior Tecnico/INESC-ID

* Conference Treasurer:

   Duane Rettig - Franz, Inc., ALU Director

* Publicity Chair:

   Daniel Herring - ALU Director

* ALU Treasurer:

   Rusty Johnson - TASC, Inc., ALU Director

Program Committee:

~~~~~~~~~~~~~~~~~~

* Antonio Leitao - Instituto Superior Tecnico/INESC-ID, Portugal

* Alex Fukunaga - University of Tokyo, Japan

* Charlotte Herzeel - Vrije Universiteit Brussel, Belgium

* Christophe Rhodes - Goldsmiths College, University of London, UK

* Didier Verna - EPITA Research and Development Laboratory, France

* Duane Rettig - Franz, Inc., USA

* Giuseppe Attardi - University of Pisa, Italy

* Jeff Shrager - Symbolic Systems Program, Stanford University, USA

* Joe Marshall - Google, Inc., USA

* Julian Padget - University of Bath, UK

* Keith Corbett - Clozure Associates, USA

* Kent Pitman - PTC, USA

* Manuel Serrano - INRIA Sophia Antipolis, France

* Marc Feeley - University of Montreal, Canada

* Marie Beurton-Aimar University of Bordeaux 1, France

* Mark Stickel - SRI International, USA

* Matthias Felleisen - Northeastern University, USA

* Scott McKay - ITA Software, USA

Contacts:

~~~~~~~~~

* Questions: ilc10-organizing-committee at alu.org

* Program Chair: ilc2010 at easychair.org

For more information, see http://www.international-lisp-conference.org

I'm curious about when are "clojurers" going to join forces with the rest of lispers on these kind of conferences. Specially since this one is collocated with SPLASH 2010, maybe it isn't such a bad idea to start doing it now :)

Yet another Clojure + Compojure + Google App Engine post

2010-05-31T13:19:00.001+01:00

Have I mentioned I'm kinda late to this show? Well, better late than sorry. Anyway, I think you can't have too much of these, specially in times where google application engine (GAE) is growing and so is Clojure.

The scenario was like most others: I wanted to build my webapp using GAE and clojure, and also stay on the bleeding edge of each technology. So lets see what I've got. I started by taking a look at the "literature" available as blog posts and group discussions (such as this and this).

I installed lein, and created a standard project.clj (By force of habit, forgot to use "lein new", but it's practically the same). I added the following directives on defproject:

:source-path "src"
:compile-path "war/WEB-INF/classes/"
:library-path "war/WEB-INF/lib/"
;; namespaces will trigger AOT compilation to bytecode. you want this!
:namespaces       [ginasiosunderground.servlet]

My goal is to produce a war directory, to upload to gae servers. So I told lein to compile to war/WEB-INF/classes, using war/WEB-INF/lib as a classpath dir. (More to come later on that :namespaces directory). The final project.clj is the following:

You can see the list of dependencies. I'm using the latest clojure 1.2, compojure 0.4, hiccup 0.2.4, a ring 0.2 (specifically adapted to use with gae servers) and appengine-clj (from r0man's fork). I also use clojure-json (to make my life easier on jQuery client side ajax calls). I'd like to mention the libraries from google. As of now, a default lein setup can't find those up to date versions (they are available, but I didn't want to toy around with maven repositories to make the magic happen). So I got those jars from the gae java sdk (download it from gae's site) and placed them on my war/WEB-INF/lib folder. Now be ware, you should have a copy of them somewhere else, because each lein clean you do will delete them from the lib directory! (eventually I'll get around to automate this process. I also know there's a "lein gae" plugin somewhere, but haven't tried yet, I'm not sure if it does what I want). One last thing about lein, you'll have to remove commons-fileupload-1.2.1.jar from your lib dir, otherwise gae will complain about it (it did, for me!).
Let's move on, then. my-app.servlet loads some stuff I need for the servlet, generates an HttpServlet class, and imports some classes so that I may load a jetty server and use gae's services (datastore, etc) in interactive mode (read, using SLIME with swank-clojure). Here's the namespace:

The magic that's left is, after issuing the defroutes and defservice command, the following (thanks to r0man for the tip!):

So here we are, running a webapp that can be deployed to gae (if you have read the linked posts, you'll see that you also need a web.xml and a appengine-web.xml to deploy, and you'll also need to comment out the ring.adapter.jetty from your namespace - gae already has its own application server :). Now by running the local server via (start-server) and not via gae's dev_appserver.sh shell script, you can happily work, on slime, redefine your servlet actions and refresh the browser to see your changes. Yay!
One problem, though. How do I use google's services? appengine-clj to the rescue, browse the code (see the test-suite for examples, specially on how you can test them locally using macros like appengine-clj.users/with-user-info). I was basically a happy camper until I got to try out user logins. GAE has this method, createLoginURL and createLogoutURL (applied to a UserService and a destination page string), that takes us to a dummy login screen. To be able to use this Service, I wrote the following code (loaded before starting the jetty server!):

But while this compiles perfectly, I noted that the login screen is not loaded, i.e., clicking the login link yields a 404 page. Not good. But it should be loadable, since the dev_server has it. Google's documentation isn't much help here, and I can't seem to find how on earth I access that screen (I need it to try out administration pages!). So feel free to drop a comment if you happen to know the solution for this!
To conclude, I'm not going to publish my application as a "blank example", as there are already some around. That said, if you have a question about this setup, just ask and I'll be as helpful as I can!

On Clojure error messages, II: ask and ye shall receive

2010-05-28T14:44:00.001+01:00

I now realize my previous post carried some strong emotions, most of them not too good, and I want to humbly apologize for it. Let me state (again) that I'm enjoying working with clojure (for the most part, at least).

I wrote about the useless (wrong?) error messages, specifically when passing a list instead of a vector (ISeq) as arguments for a defn. After experimenting a bit, I was even more confused: (defn x () 1) complaints about the integer, so does (defn x () [] 1) and (defn x () [1]), clojure just doesn't know how to create an ISeq from that integer. I now understand this makes sense because you can specify multiple arity functions, like this:

(defn x

([] (x 0)

([x] (+ x 1)))

So in this case, if a vector isn't found where the arguments are expected, clojure complains, as he can't make an ISeq.

But none of this matters anymore (to the language user). Stuart Halloway (author of the book that I used to learn about Programming Clojure, which by the way I advise you to read, if you haven't already) was almost to fast picking up this issue! I have just received his tweet with a fix to this problem: clojure (the snapshot version) now detects if the arguments are a proper vector, and if not warns us with a much clearer message: "Parameter declaration arg1 should be a vector". This should satisfy all with the same issue I tackled on the previous post; more advanced users dealing with multiple arity defns should already be aware of the syntax, so it's safe for both cases.

This was a positive outcome in more than one way. Not only clojure got a bit better, but I got reminded of how I should had followed a much nicer "netiquette". I made a public complaint, and Stuart was impeccable to pick it up, hacking a fix and letting me know about it (everyone should learn from him, including myself!). But instead of posting my complaint (again, sorry if my tone was to harsh), I should had (1) ask on a discussion list or on other social platform about this; (2) take a look into the source code, see where the issue could be fixed, write a patch, and submit it to approval to the developers - alternatively, take the oportunity to learn how to do those things, and at least let the maintainers know about it, so a bug report is issued. This is the organization that most projects require to evolve properly.

On Clojure error messages.

2010-05-27T12:51:00.001+01:00

I know. I'm late to this train. And there are lots of resources to clarify this. But I have to manifest myself somewhere (and twitter's char limit won't do this time).

Here's the deal. I know I've written a fair share of Clojure lines of code. But I come from the land of the ancient parenthesis (specifically, Scheme, Common Lisp). So whenever I try to teach clojure to someone, it's inevitable to tell them that it's a Lisp, so it's ok to see those rounded delimiters everywhere.

This is where I start by asking them to define a simple function, and instead of defun/define, use defn. Something like this will do:

(defn xpto ()

(+ 2 3))

Simple enough, right? Wrong. (I know it's wrong, now, off course). But for the beginner, the first thing to do is to read the error message. So let's take a look:

Don't know how to create ISeq from: clojure.lang.Symbol

[Thrown class java.lang.IllegalArgumentException]

Yeeeah, isn't that obvious? Clojure wants to make a sequence and I'm trying to use a symb... wait, what?? the only symbol I know I'm using is xpto, and that's about right on the syntax! The Java exception isn't much help either. What argument? The ones I'm not using on the functions (but weren't those parameters?) Or the ones I'm using on the function call (but 2 and 3 are numbers, not strings!! *help*!) Ok, breath. let's take a look into the stack trace, it is usually helpful on java, at least to trace where the error is:

Backtrace:

  0: clojure.lang.RT.seqFrom(RT.java:469)

  1: clojure.lang.RT.seq(RT.java:450)

  2: clojure.lang.RT.first(RT.java:538)

  3: clojure.core$first.invoke(core.clj:53)

  4: clojure.core$sigs$asig__3331.invoke(core.clj:202)

  5: clojure.core$sigs.invoke(core.clj:214)

  6: clojure.core$defn.doInvoke(core.clj:268)

  7: clojure.lang.RestFn.invoke(RestFn.java:495)

  8: clojure.lang.Var.invoke(Var.java:381)

  9: clojure.lang.AFn.applyToHelper(AFn.java:180)

10: clojure.lang.Var.applyTo(Var.java:482)

11: clojure.lang.Compiler.macroexpand1(Compiler.java:5215)

12: clojure.lang.Compiler.macroexpand(Compiler.java:5270)

13: clojure.lang.Compiler.eval(Compiler.java:5338)

14: clojure.lang.Compiler.eval(Compiler.java:5320)

15: clojure.core$eval.invoke(core.clj:2366)

16: swank.commands.basic$eval_region.invoke(basic.clj:47)

17: swank.commands.basic$eval_region.invoke(basic.clj:37)

18: swank.commands.basic$eval801$interactive_eval__802.invoke(basic.clj:66)

19: clojure.lang.Var.invoke(Var.java:365)

20: myproject.core$eval3896.invoke(NO_SOURCE_FILE)

21: clojure.lang.Compiler.eval(Compiler.java:5353)

22: clojure.lang.Compiler.eval(Compiler.java:5320)

23: clojure.core$eval.invoke(core.clj:2366)

24: swank.core$eval_in_emacs_package.invoke(core.clj:94)

25: swank.core$eval_for_emacs.invoke(core.clj:241)

26: clojure.lang.Var.invoke(Var.java:373)

27: clojure.lang.AFn.applyToHelper(AFn.java:169)

28: clojure.lang.Var.applyTo(Var.java:482)

29: clojure.core$apply.invoke(core.clj:536)

30: swank.core$eval_from_control.invoke(core.clj:101)

31: swank.core$spawn_worker_thread$fn__453$fn__454.invoke(core.clj:300)

32: clojure.lang.AFn.applyToHelper(AFn.java:159)

33: clojure.lang.AFn.applyTo(AFn.java:151)

34: clojure.core$apply.invoke(core.clj:536)

35: swank.core$spawn_worker_thread$fn__453.doInvoke(core.clj:296)

36: clojure.lang.RestFn.invoke(RestFn.java:398)

37: clojure.lang.AFn.run(AFn.java:24)

38: java.lang.Thread.run(Thread.java:637)

Read it all? Want your 3 minutes back? (ok, 30 if you actually went looking for documentation on the API, the source of Compiler, core.clj or worse if you went looking for slime's or Thread :) ). Sick, isn't it? not a single line is useful for our cause.

I'm afraid I have no good news for you, except to tell you what the problem was and hope you'll never forget that. I know I didn't. (until I swapped to a bunch of different programming languages, including common lisp, for a while, and returned with my auto-pilot coding style turned on :) ).

The problem is that clojure expects a sequence of parameters, not a list. And thus, the proper syntax is to use [] instead of (). I know, a list is not a symbol. Or so I thought... I could even go theorize about () being translated to nil in compile time, and so it was a symbol. But evaluating (symbol? nil) quickly clarifies that for us...

Pledge - please, oh please start adding useful semantic to clojure's core language errors... The language entry barrier will lower significantly, I'm sure!

Aperture: scripting takes you where automator doesn't

2009-08-20T12:35:00.001+01:00

Like some unfortunate people, I've suffered from data loss. Mainly by distraction and a trigger-happy finger on the delete key. So I've embraced backup strategies, and my last acquisition was a Western Digital MyBook World Edition with 1TB hard disk inside. I keep it connected to my router, so it's always available from a wireless laptop in the house. I'm happy and comfortable with the setup, so the next step is, no surprise, figure out what I'm going to put there. And, most importantly, how I'm going to do that.

My first use case was my photography collection. I've recently started using Apple's Aperture to handle my pretty basic photography sets. I don't own a SLR, yet, but I have a Canon Powershot S50, which allows to manually configure the machine to take some interesting photos. And it shoots RAWs (albeit in a Canon proprietary CRW format, that can be batch-translated to DNG using the freeware converter from Adobe). So after my photo workflow I end up with an hierarchy with folders (for project types like "Trips and Events", "Experimentations", "Personal", "Work", etc), inside which i have projects.

What I wanted to achieve was a threefold backup: Save the project itself (so that i may open it in other aperture libraries, both on my macbook pro and my desktop mac); Save the masters, raw files; Save the versions, in JPEG. I figured I'd try Apple's little Marvin, the Automator, to help me easily do it. I got excited to see that there were some export actions available for use in workflows! That, however, proved to be insufficient, as there is no action to get the currently selected project. This was the first applescript i used on the workflow, to get just that, and place in a variable to pass on later. The code looks something like this:

tell application "Aperture"

activate

-- identify currently selected image(s) project

set x to selection

set x to item 1 of x

tell library 1

set ap_proj to (get value of other tag "MasterProject" of x)

end tell

I was pleased with the result, as the MasterProject tag on the currently selected image would tell us the project id, not its name (like we would get if we selected from the entire project list on aperture). But then i found out two drawbacks:There's no way to export a project bundle directly via an action. There's also no way to get the project name to pass as input to the export versions/masters actions, as they require a list of images. So after a visit to the Aperture dictionary on Script Editor (just open the application with it, and browse around), i found out i could export a project bundle via applescript (great), and produce a list of the images in it (also great). But not everything is that easy, as I found out that not only the Export actions don't like to accept input from a "run applescript" action (they require a variable to be set and then got, for them to receive data flow), but when the script is saved (or exported to a plugin), the flow is broken and the export actions cease to work!

I ended up giving up the entire workflow, and scripted the whole thing. I only kept the automator workflow file so that i could easily add stuff like growl warnings. You can download it from here and give it a try. Oh, and if you know an easy way to find out the folder a project is in, to put the bundle in that directory, please, leave a comment or tweet me!
With this script (with a small adaptation) it's also easy for me to loop through all projects and perform a complete backup of the library. This sure beats the workflow-actions way, as there's no amount of variables and loops that can make you run through a list of projects. For these reasons, I believe Automator needs a 3.0 update. Maybe Marvin could take some anti-depressives, and leave the paranoia behind? :)

Mouse-driven love to buyers on Firefox

2009-06-15T01:30:00.001+01:00

A while ago I was amazed by the truly great work that the Mozilla Labs team did on their project Ubiquity. If I ever thought about a web-based operating system, this all-in-one launcher is anything I'd ever need: fast, useful and easy to extend. Right now the development is going towards a better integration with the "normal" browser usage, melting it's UI with the amazing toolbar.

There are, however, a huge piece of the user pie that aren't that fond of writing, or using the keyboard, unless absolutely required. I've heard about some prototypes with a mouse-driven ubiquity, but as far as I know that idea never took off. There's still time, though, and all these firefox extra-functionality are pretty much in early development stages.

So it was half surprised and half amazed that I watched as two of the master-minds behind Ubiquity (Aza Raskin and Atul Varna) releasing another revolutionary project from their labs: Mozilla Jetpacks. Jetpacks are a way to extend the browser by using a set of Javascript instructions. The main motivation is that up until now we had to grasp the XUL doctrine just to make a popup message greeting the user, and it's easily seen that the amount of Web developers that already tamed some Javascript is growing by the minute.

Wait a bit, I was babbling something about mouse-lovers getting some attention, and I was just getting carried away... That's true: Jetpacks architecture allows certain Javascript objects/functions to interact with certain browser features (statusbar, menus, making HTTP requests, etc.) One of the newcomers into the recent feature-list on its API is called the Slidebar. It's just like a Firefox sidebar, but it stays hidden unless we hover on a small arrow on the top-left corner - then it slides and shows us icons for each slidebar jetpack we have installed. The first slidebar I've seen was the Wikipedia, allowing users to select some text, click on the Wikipedia icon, and see the mobile Wikipedia page for the selection on the slidebar. Pretty sleek - specially if you don't want to fire up Ubiquity, type "wi" and watch the preview window assuming the "wikipedia it" completion and showing up the results inline. And I know there's people like that :)

I quickly thought to myself that most reference websites would be much easier to access there (as opposed to, e.g., the search-bar, where I find it clumsy to switch between search-engines). But I've also thought it would bring some advantage to Ubiquity in one simple aspect: user input. We're confined to a single sentence on Ubiquity. And despite being oh-so-cool to parse natural language and extract the right meaning out of it (because it is!), some things take more than one sentence to say clearly, and there's no way around it.

I thought about the command I've made for Ubiquity to add financial transactions to Buxfer. The syntax I accept on it is "BUXFER-SPEND <amount> in <description>", where <description> may be a set of words with arbitrary length. My problem with this is twofold. First, it's not easy to extend this with other parameters (like the date of the transaction, its tag(s) ...). Then, a technical issue makes the command keep presenting one completion suggestion per word on the description. (it has to do with not being the direct complement, i think. And doing a manual parser is *not* fun to developer on a sunny afternoon!) And that just sucks...

So my idea was to take advantage of a slidebar to show a form, already filled with the selected text on the browser. The form has the focus on an amount text field, and a submit button (extra fields are easy to add later. Just ask me if you need any particular data to be posted, and I'll look at it). I picked up my ubiquity command code out of my shelf, and used most of it to connect to Buxfer (using its API). Besides the command itself, I had to use a different message display mechanism. I was thinking about giving Yip a try, but it turns out the notification system on Jetpack is perfect as it is - Growl and toaster goodness for everyone (don't know about linux, feel free to show me screenshots with their standards notifications!). Sorry Abi, I'll keep Yip to use on web apps, for now :)

One thing that slowed me down a bit was to find out the Utils object we have on Ubiquity is not on the Jetpack API. But that's pretty much ok, since we have the jQuery framework at our disposal. This way, instead of using the JSON decoding mechanism i used on Ubiquity, I used the jQuery ajax() and getJSON() functions to return native objects instead of JSON strings. And all is good in the world!

Some conclusions out of this fun experiment - great frameworks allow doing great features easily and, more importantly, they allow to have fun when programming them.Just the Bespin being there for anyone to give it a quick test-drive is enough to capture youngster's attention (I just keep trailing back the text to my Emacs buffer using the It's All Text! addon). So I give 10/10 to the Jetpack/Ubiquity efforts (taking into account their early state of development, obviously). Also, I ask those projects developers to unify their support as much as they can. Ultimately, I think an Ubiquity command should be one of the ways a Jetpack could extend the browser!

Without further ado, you can find my command, Buxfer Jetpack here. It should "just work" if you have Firefox with Jetpack installed. If you run into any issues, tell me, so that I may learn what they are and how to fix them. And prevent their repetition!

It's running late, but it has sure been fun to do this. The best part? I may be using this jetpack myself, complementing ubiquity's command. And that makes me being feeding myself with my own dog food.

ELS 2009 - my impressions

2009-06-01T21:10:00.001+01:00

I've finally got to attend an international Lisp meeting — one more item crossed off the checklist!

The European Lisp Symposium 2009 was held in Milan, Italy, during the 27-29 of May. The program featured some interesting presentations, and while I'm not going to go a complete overview of them all, in this post I'll go over the bits that got into my head the most. This means, don't be offended or ignored if your work (or the work you were expecting to read about) isn't mentioned here. It probably means my mindset isn't properly configured, yet :)

The first day of the conference was just the reception, where I got to meet great people. Among the entire conference I got to meet people I've already read about in the web-o-sphere such as Stellian Ioanescu, Scott McKay, Mark Tarver, Nikodemus Siivola, Pascal Constanza and Christopher Rhodes. I also got to know some nice people I didn't know, including Didier Verna, Jim Newton, Claus Brod and Edward Dogde. António Leitão, my PhD supervisor and program commitee chair, introduced me to Mark Feeley, from Termite and Gambit Scheme, who told me about a Gambit package, JSS (a multithreaded javascript compiler), which I must try when I get back home. Most of all, despite some ideological divergences between some of the participants, the mood was quite pleasant, and joyful. After all, Lisp is supposed to bring happiness to those who use it.

The second day started with a Scott McKay talk on his life's experiences, and mistakes. He has grown a pretty strong aversion to incremental code fixes to correct fundamental problems. Pretty much all he's done made him believe that full rewrites gives us the opportunity to think better about the design, and that we cannot try to simplify the code by extending it. He is also concerned about the current "big thing": concurrency. May languages are addressing it, and Common Lisp is getting behind each day that passes. Clojure tackled it, but it carries the known set of burdens. Scott recognizes the great one-man effort, but wonders if it's easier to improve Clojure or to design a new Lisp and get it right from design. This could rapidly solve, for instance, the namespaces Common Lisp problem (instead of having symbols, packages, labels, macros, symbol-macros, etc, etc (Pascal counted 9 different namespaces), we could have one single namespace). The same goes to the type system, *everything* should inherit from the same root object type.

Mark Tarver gets an almost similar position, but he "strayed away" from Common Lisp, and went looking for answers on both Scheme and Python. Because he couldn't find them, he designed Qi, a language that is supposed to offer what he believes to be the best language features.

I found the presentation on hygienic macros for the unhygienic world quite motivating. By using a few helper macros, Pascal Constanza demoed how we can emulate the former on Common Lisp, which is quite useful, specially if you come from the Scheme world. All at the expense of a few extra language constructs, but not too distracting ones.

Charlotte Herzeel presented an interesting approach to implement Software Transactional Memory (STM): she implemented a (limited, not feature-complete) Scheme interpreted within Common Lisp. Doing this allowed her to have access each memory access point easily (she only has to address cell and vectors accesses. The thing is, by having her interpreter, she easily tapped to the code segments where data access is done, via reflection. So it becomes easy to experiment with STM algorithms.

I found out the industry is not sticking to widespread CL implementations. At least three different companies (mostly related to the CAD/graphics world, but I think it's a coincidence) rely on SKILL, or some adaptation of it. SKILL is a small footprint Lisp-like scripting language, and that makes it appealing for many industrial domains. However, it appears to be quite obsolete, undocumented and unsupported. It was nice, though, to see some successful uses of Lisp.

I was kinda disappointed not to see Kent Pitman, I believe he canceled at the last minute. I was also looking forward to see some Clozure CL people there, I'd like to hear about what they have in mind about Cocoa and iPhone development. They already have a great Objective C bridge, but as a Common Lisp, things are entitled to be much easier, specially in order to produce competitive small products (there's a big niche for small simple applications on MacOS, due to Apple's design principles)

Marco Antoniotti was a great host on Milan. He organized a great banquet at Osteria del Treno, where we all got to share some more insightful comments about the entire conference and more. The day after that, Marco wore the cicerone cap and took everyone that was still in Milan for a great Futurismo Exhibit, where we got to learn some amazing bits of the Italian cultural inheritance (definitely not my area of expertise, but it was quite interesting to try to interpret some works and sell my ideas to the guide, despite failing most of the times :) ). That afternoon we visited the Duomo, a massive cathedral on the center of Milan. The most impressive thing for me was to realize the wow-factor was even greater from inside of it (on it's roofs), than from the outside!

All in all, it was a memorable experience, everything went great (i wasn't even bugged with the wireless access difficulties that so many people were forced to have, since i had the eduroam credentials already set up on my machine!). Next year it'll be in Lisbon, so I'll be attending to. Meanwhile, I'll look forward to go to Italy again on July for the 6th European Lisp Workshop (within ECOOP).

Ubiquity in Portugal's Portuguese

2009-03-05T22:34:00.000Z

I've been keeping up with some of the stirring around Ubiquity development, and one aspect that caught my attention was the adaptation of that amazing interpreter to multiple languages. I've commented before on Aza Raskin's blog about the implications of using Ubiquity in native portuguese. Mozilla's Mitcho Erlewine and Felipe Gomes have brought up the issue again. This time things appear to be moving on in Mozilla Labs for Portuguese language in Ubiquity, which is great! But please, hear me out first for a few small — but relevant — points that need to be considered. In the remainder of this post I'd like to clear them out. Note that these will be personal opinions, and I'm not a linguistics expert, only an interested software engineer.

Felipe's entire post is clear and understandable. But his ideas work for the Brazilian variant of the Portuguese language. Being a native Portuguese (and while not having any kind of grudge against non-native Portuguese), it's somehow odd (even repulsive, pardon me!) to read some of the Brazilian syntax — it's just looks badly written. One simple example is the "gimme a map" command, that Felipe translates to Brazilian as "me dê um mapa", and native Portuguese would translate it as "dê-me um mapa". But this simple example presents other cultural divergences, like the friendly/respectful tone. The English expression "give me" can be translated into "dá-me" or "dê-me" (friendly/respectful, in that order). And the same goes to the command verb "map", that would be translated into "mapeie/mapeia".

Felipe's offers the great suggestion of detecting the word's root, e.g., "map" in "mapeie". And this will work for almost every verb. Except some crucial ones, that are irregular! (I honestly hate grammar irregularities...). Take the verb "to go" (e.g., can be used in a command to go to a specific page/tab/section/<somethingelse>). So one could say "go to gmail". In native portuguese, Felipe's suggestion fails in that the same verb translates into "ir", and is used in a commands such as "vai para o gmail", or "vá para o gmail". Note that not only these commands cannot be easily traced into their infinitive form ("ir"), but both possible tones are different (the accented "a" in "vá" is very much relevant).

I'm really sorry to be a spoil-sports, but this is won't work so simply. Obviously, the set of irregular verbs are finite, and we could add the correct usages to a syntactic rules table. I expect the final code to be optimizable into a feasible speed of interpretation. But I'm not so sure that this rules can be, as Mitcho so desires, reduced to a common denominator to all languages (again, don't get me wrong, I'd love that to be possible, I just don't have a useful knowledge-base of other languages to believe that to be feasible).

And this is why I'm so fond of using some webapps' names as commands, even in portuguese. "twitter <message>", "google <query>" or "wikipedia <expression>" will be understood perfectly in Portugal (and Brazil, for that matter). I know, Ubiquity's future is about natural language interpretation, whatever idiom you may prefer. And that's really something special. But I can't stop but to feel that every language is either going to require a complex interpreter suited for itself, or suffer from aggravated syntactic-sugar syndrome.

If any point I've covered can be countered somehow, just consider, in following blog posts/projects/etc to remember to identify what Portuguese idiom are you referring to when you discuss grammar rules. Again, it's not a matter of any kind of discrimination, but one of correctness, and everybody deserves that.

Thanks for reading!

Lisp macros, eval and packages

2009-01-28T15:44:00.001Z

Today I spent about an hour trying to find out the source of a bug in the code for my PhD prototype. To make the context short, I need to have a class definition to be stored, and loaded afterwards. And I was using Gary King's time-saver metatilities:defclass* to reduce the required coding. The problem was when I wanted to load such a definition in a generic method, I was evaluating the definition expression - one of the acceptable uses of `eval', correct me if I'm wrong, is to interpret something that was just read (in this case, from a serialized string, from other servers). The code in question was something like this:

(defmacro load-data (name attributes)
  "Defines a domain concept, using metatilities:defclass* syntax."
  `(progn
     (metatilities:defclass* ,name (data-item)
       (,@attributes)
       (:metaclass profiled-metaclass)
       (:name-prefix  ,(format nil "~(~A~)" name)))
     (info "Data type loaded: ~A" ',name)))

(defmethod load-definition ((def data-item-definition))
  (eval (macroexpand `(load-data ,(definition-name def)
                                                     ,(data-item-definition-attributes def)))))

What I was getting as a result is that the auxiliary methods created as accessors by defclass* weren't being imported in my packaged, but in CL-USER. Hence, lots and lots of compiler warnings and subsequent errors when trying to use those accessors. I fiddled around, tried to see if I could remove the eval, but the solution, so obvious as I know can see, is quite simple. Just place your evaluated expression after setting the package. The fixed method is the following:

(defmethod load-definition ((def data-item-definition))
  (eval `(progn (in-package :dshow)
                ,(macroexpand `(load-data ,(definition-name def)
                                          ,(data-item-definition-attributes def))))))

Hope you can see this before spending too much time on a similar bug!

Quick Emacs nicity

2009-01-26T19:03:00.001Z

I can’t remember from whose configuration I got this piece of code. But it makes my lispy screen a little bit cuter, by turning all “(lambda” into the greek character. I tested on aquamacs, and on the Windows GNU Emacs, and it works. If anyone knows the original author of the idea, send him/her my thanks :)

;; Make lambdas appear as the character » (saves space!!)
(defun pretty-lambdas ()
  (interactive)
  (font-lock-add-keywords
   nil `(("(\\(lambda\\>\\)"
          (0 (progn (compose-region (match-beginning 1) (match-end 1)
                                    ,(make-char 'greek-iso8859-7 107))
                    nil))))))
(add-hook 'lisp-mode-hook 'pretty-lambdas)
(add-hook 'emacs-lisp-mode-hook 'pretty-lambdas)

On iPhone SDK and its pretty icons on tab bar items

2009-01-02T13:26:00.001Z

After having made some experimentation around the iPhone SDK, I have now a simple - but useful - native application. I focused on the functionality, and up until now, I had a tab-bar with no icons. But that's not the Apple-way to present things. iPhone apps are pretty, and have simple but elegant icons with a gray tone that turn blue when selected. So i looked upon a library/bundle for some of those standard buttons, and found... none! Ok, no worries, let's find out how to make them. Wait. There isn't a specification for them! Yes, it's a shame, but Apple's iPhone SDK documentation is still a bit unfinished. The good news is that what is done, is easy to read and helpful. I ended up looking at other apps resources, found out they were 32x32 png images, and figured they had to have at least an alpha channel outside the image itself. So I fired Gimp, made a plain brush doodle and removed the background white, saved and tested. And it simply works! All colors are turned into the desired gray tone, and automatically swap to a smooth blue with a top glare when selected. Conclusion: the trick is to experiment. Chances are that what you want to do is *that* simple!

Milky love for lispers

2008-10-25T22:20:00.000+01:00

I've just updated my "Remember The Milk"-talking lisp library. Now you can access their API with a simple CLOS-based object model.

You can fetch the latest version by direct download, SVN and/or by asdf-install from the project page, where I have the important updates:

Recent Changes

* Added an object bridge to make the API even more integrated with the language.

* Methods and functions were coded to be easier to manage a local state of the RTM user data.

Future Work

* Improve synchronization and bandwidth usage by fetching only recently updated data;

* Provide offline work for the local data

Having a cow aiding getting your things done

2008-10-12T14:27:00.001+01:00

I have finally stopped to upgrade my Remember the Milk (RTM) usage to something more GTD-ish (for David Allen’s “Getting Things Done” approach). I already knew about RTM’s organizing abilities. Succinctly, we can have tasks in our inbox, in lists (hard to move from, usability-wise), and have them assigned one or more tags, easily exchangeable and searchable. What I was looking for was a way to have my tasks split into contexts (like calls to make, errands to do, things to do at the computer, things to see, read or listen). Also, I need to see what I can do next, what I have to do someday, and things I must hold on pending, waiting for something/someone. With these two axis for plotting my tasks, I can add stuff to RTM as they come up in my life (via many different ways like their website, Twitter, quicksilver, a Dashboard widget, iPhone...), and they will get placed into Inbox. At the start of the morning, I have to process that list, placing tasks into a proper context List (I prefix those with a @, to visually identify them at a glance). I also tag each task with a “next”, “someday” and “waiting” tag. The final visual output is clean enough for me to feel organized:

Let’s get the webpage hacks discussed first. It’s simple enough, but important for me. The default tag strip wouldn’t do it for me. So I got this amazing Greasemonkey script, called A Bit Better RTM. Three simple features that make all the difference!

Step one, projects. I followed many advices at the RTM forum, and created a list called Projects, where I place project titles (prefixed by a dot, for visual identification), and tag them with a “p-PROJECTNAME” tag. This way, I can easily click that tag and access all tasks (from one or more contexts). It’s really handy.

Step two, smart lists. These are simply saved searches, and we have a decent set of search operators to use. Here’s some I have installed:
• ->Next: (tag:next AND NOT dueAfter:"1 week of today") OR dueBefore:"1 week of today"
• ->Someday: tag:someday
• ->Waiting: tag:waiting
• ->Tickler: NOT due:never
• PhD: tag:PhD OR tagContains:phd
• Fun: tag:entertainment OR tag:game OR tag:movie OR tag:night
• unTagged: isTagged:false OR (NOT (tagContains:"next" OR tagContains:"someday" OR tagContains:"waiting"))

Tickler is a must see every day, to keep myself from forgetting due tasks. Phd looks for tags like p-phd-thesis, thus looking for all projects related to my Ph.D.. unTagged is a reminder tool. It often happens that I move a task from inbox into a given list, and forget to tag it (all tasks should have a next/someday/waiting tag on it).

Don’t forget that tasks can easily be assigned a priority with a 1-4 key press. Also, I have locations set up so that they appear in the task cloud. This way I can look at all the calls I have to make at the office, for instance.

So with this I feel almost at ease. But I’m kind of a control freak, and I like to add tasks in the easier way I can. After having played with Firefox’s ubiquity to write Twixfer, I started jotting down a quick way to add a task to RTM. The initial ubiquity command was usable, right until it popped up a new tab with a RTM task properties form. I didn’t need that. But someone felt the same, and spared my the effort. rtm-v2 is a command set that logs in, and gives proper feedback of your RTM lists and tasks. If you’re into speed task jotting, it’s worth a try.

That’s my rtm tips for today. Do it like me? Have a better system? Drop me a comment!

Love Buxfer. Use Twitter. So why not Twixfer?

2008-09-07T20:25:00.001+01:00

Today I sat down to find out a different use for Javascript. Browsed through my everyday task list, found out a couple of candidates, and targeted Buxfer transaction creation. I'm also growing fond of Ubiquity, for the Firefox browser (and is now the most important reason I'm not moving to Google's Chrome, but I digress). This post, despite being about a Buxfer contribution, will be more about the Ubiquity development process I underwent.

So let me explain what I aimed for. Ubiquity allows to enter text controls, with auto completion in a HUD over the Firefox window. So I wanted to do something similar to what Bruno Lopes did with his BuxferSubmit Dashboard widget (for Mac OSX) - plain simple addition of a Buxfer transaction, with a description and an amount. All I have to do is (after having Ubiquity installed and running) to setup a Javascript file with a piece of code I'll explain hereafter.

The heart of a Ubiquity command is defined with the commandfunction CmdUtils.CreateCommand(argument-object). This is, in fact, the only call you have to make in that file to set things up. So what's in an argument literal object? the basic attributes, like the name of the command, it's description or the icon to identify it (quite useful!). But there are a few arguments that deserve special attention

execute. This must be a function with no arguments that is responsible for doing whatever the command is supposed to do after <return> is pressed.
preview. This is the function that is called at each key press on the Ubiquity HUD, to compute the string that appears to help the user find out what the command is going to do.
takes. A description of the words the command receives, and a type for them. Each type can be specified by hand, but several are already provided, like noun_arb_text for arbitrary text or noun_type_date for a date string.
modifiers. These are words that offer secondary semantics to the command, and are specified just like the “takes” argument elements. Think of examples like “list-birthdays” for today's birthdays and “list-birthdays for next week”. The modifier “for” would be assigned to “next week”, and a special type can be setup to parse the string and compute the date from today.

To find examples about how to use the CreateCommand, load the following URI in Firefox: chrome://ubiquity/content/builtincmds.js (obviously only after having ubiquity up and running).

That's about it. What I found out was that the presence of a preview functionality can often be more useful than the execution of the command! Think about a currency conversion, or an algebraic expression calculation. I don't want to perform anything upon a <return>, except perhaps have the result copied to the clipboard. What I really want is to have the result appear in the HUD and change dynamically as I keep filling in the command text.

Let's move on to the implementation itself. One thing I found out quickly was that the Buxfer's APi needed a username and password to be able to post a transaction. That would be the case in most web applications I know about. But this API doesn't offer, for now, any means to open a page to ask for credentials - nor would I desire that, since I'm typing a textual command to keep the input at a minimum! Ubiquity doesn't have a thorough documentation, yet, and browsing the javascript XUL files within the chrome “filesystem” of a Firefox add-on is not something I like to call user-friendly. Why is this important? I thought the command could take a peek into the Firefox's secured passwords and use the Buxfer, without the user having to care a bit about it. But then again, an exception would have to be made in order for first timers be able to use the command… I was about to set up a proxy web service in Google's App Engine when I remembered two happy coincidences:

Buxfer allows Twitter messages to be sent with the same syntax as those we can send from an API (or from a cell phone).
Twitter has a nice API, that does exactly what I was aiming for: it points it's requests to they're site, and if you're not logged in, it asks for your credentials using HTTP Basic Authentication. This way, Firefox aks you if it may remember the password, and actually does it :)

The result: I ended up with two commands, one using the Buxfer API, and another using Twitter's. But since Twitter's the most user friendly (easier and faster to use), I'm switching my transaction input to the brand new Twixfer (It's weekend, my imagination needs some more rest to come up with a better name!). Feel free to take a peak into the homepage. The source consists of a couple of files, so I thought a Google Code project would be over-killing it… But if this thing starts to get any sort of user input/contributions, I'll gladly move it there. For now, leave comments at the Twixfer site. As usual, feel free to leave feedback about anything.

[EDIT: Today I had a tip from a gentle soul about retrieving password from Firefox's session. So Buxfer API is usable by itself, now. Check the same link and update your command, if needed. It has lots of cleaning up and code re-factoring done. Nevertheless, it was fun to toy around with the Twixfer concept!]

Getting Things Done - the flow-chart

2008-09-02T23:56:00.002+01:00

Here's David Allen's graph for Getting Things Done (GTD). Though to use, but has nice results!

Moooing away in Common Lisp

2008-04-14T00:17:00.001+01:00

I'v recently found myself needing to use a task manager to make some bulk operations (to be more precise, I wanted to copy all of my tasks to a device with no built-in synchronization mechanism for those). I happen to make use of the great Remember The Milk Web application (I've previously worked with Emacs Planner, but I had to drop it in favor of a more mobile version of my life).

Enter Common Lisp. Perhaps the fastest way to do some scripting in my mind, but I've yet to make a library to use a Web-based API. Turned out it was quite clean, and in about 5 hours of work I had the whole API grokked, coded, tested and hosted in GoogleCode here. (I looked into cl.net, but found the google's approach to hosting much more featured for the fast start I was looking for. If this has enough momentum I can look into it again, with more time - the main advantage would be asdf-installability, I suppose.)

I already had experience making project "skeletons" (asdf project file, package declaration, etc.). So the hardest thing I had to do was look into the web API call mechanism and into the md5 encryption. If you don't need the detailed description, tha answers are, respectively, Drakma and Ironclad. ( I also use cl-json to parse RTM's responses into a lisp-friendly format.)

To call a Web-based method, we need to make an HTTP request, and be able to read the response. using drakma (asdf installable and loaded in the usual way), all I had to do was something like:

(http-request rtm-api-endpoint :method :post :parameters '(("param-name" . "param-value")))

With some parameter abstractions, RTM's was easily tamed. A call to one of their methods is now made with the following macro:

(defun rtm-api-call-method (method &optional key-value-pairs &key with-timeline with-authentication (format "json")) "Calls `METHOD'. - Optionally passes pairs of strings in `KEY-VALUE-PAIRS' in the form of (`PARAMETER' . `VALUE'). - Keyword `WITH-TIMELINE', if not null, allows the method to be called within the `*CURRENT-TIMELINE*'. - Keyword `WITH-AUTHENTICATION', if not null, allows the method call to be authenticated with a valid `*RTM-API-TOKEN*'. - Keyword `FORMAT' is one of \"json\" (the default value) or \"rest\", and specifies the server reply format." (declare (special *current-timeline*)) (let* ((parameters `(("api_key" . ,rtm-api-key) ("method" . ,method) ("format" . ,format) ,@(when with-timeline (with-timeline `(("timeline" . ,*current-timeline*)))) ,@(when with-authentication `(("auth_token" . ,*rtm-api-token*))) ,@key-value-pairs)) (api-sig (compute-rtm-api-sig parameters))) (multiple-value-bind (result) (http-request rtm-api-endpoint :method :post :parameters `( ,@parameters ("api_sig" . ,api-sig))) (let* ((response (json-bind (rsp) result rsp)) (stat (assoc :stat response))) (cond ((string= (cdr stat) "ok") (rest response)) ((string= (cdr stat) "fail") (let ((err-info (cdr (assoc :err response)))) (error "RTM error code ~a: ~a~%" (cdr (assoc :code err-info)) (cdr (assoc :msg err-info))))))))))

With this (rather big, for the sake of readability, or mayhap due to my lack of experience in mental pretty printing) function, I define a call to an RTM module simply by coding:

(defun rtm-api-tasks-complete (list-id taskseries-id task-id) (rtm-api-call-method "rtm.tasks.complete" `(("list_id" . ,list-id) ("taskseries_id" . ,taskseries-id) ("task_id" . ,task-id)) :with-authentication t :with-timeline t))

The only call in the first function definition that cannot be easily guessed how they work is the compute-rpm-api-sig-parameters. This is the function that performs the algorithm required by RTM to encode all parameters and guarantee authenticity in each request (along with a masked API key). Basically it does some mambo-jumbo concatenation with each parameter's name and value, and then performs an MD5 to get the equivalent hash code. Easy, right? But I'm a lazy programmer. That, and I truly believe there's no point in reinventing the wheel, so I looked for encryption functions in Lisp, and I found Ironclad, a collection of most of these functions in Lisp. So producing an md5 functions was as easy as the following:

(defun md5 (string) "MD5 uses ironclad to encode `STRING' into an hexadecimal digest string." (ironclad:byte-array-to-hex-string (ironclad:digest-sequence :md5 (ironclad:ascii-string-to-byte-array string))))

And that's about it. The rest was toying around, the API works perfectly (kudos to the RTM dev team!), and I was thrilled to have my tasks exported in a jiffy. If you want to find more info, feel free to snoop through the code (it's quite easy to grasp, I think, and relatively small), in the Google Code site: http://code.google.com/p/rtm-lisp-api/ . So now, what would you, gentle reader, do with this API? Drop me some comments with your ideas.

Until the next time!

Cocoa development, using a Lisp bridge

2008-03-13T19:20:00.001Z

Let's get a bit away from my PhD for a while. In my spare time, I've been looking at some graphic libraries to make some applications. In the past I've looked for Operating System portability for those apps. I've also delved into the Web world for interfaces. Right now, I've settled for making pretty looking gui's. If you add my new acquisition - a Macbook Pro, my first mac ever! - and I obviously had to try to make a MacOS Cocoa application. They're fashion, they're usable, they mingle with the rest of the desktop manager, and I was just tempted to try to use Lisp to make something like it.

Fortunately, this wasn't so hard, has OpenMCL has a great step by step tutorial (to be taken with a few glasses of water and with the Cocoa / Interface Builder tutorials made available by Apple). This tutorial has detailed instructions of how one can build the Interface with the WYSIWYG tool that comes with Xcode. Then it tells you how to setup the connection points between the interface and the application code. These are called outlets, for the variables in the code that represent a given object (so that you may, for instance, retrieve the text in a text field) and actions, for the functions that are called when something happens in the interface (e.g., user click in a button). After creating the necessary objects (basically you must make them belong to a specific metaclass in order for them to be exported as an ObjectiveC class), all you have to do is call a function that binds together the compiled code, the Interface files (in the nib format) and places a nice application bundle in the directory of your choice. It can't get any easier, you think. And yes, you may code interface changes in you lisp code. But it's extremely easier to follow in Interface Builder guidelines and helpers... You just can't go wrong!

You may find the tutorial at the OpenMCL wiki: http://trac.clozure.com/openmcl/wiki/AppleCurrencyConverter

User interface paradigms for data search and offline data access

2008-02-01T18:29:00.001Z

I've stumbled upon this interesting and illustrated article on data lookup user interface design patterns (on both web and desktop applications, finally we seem to be starting to forget there's a difference!):

Here are two design paradigms for handling large amounts of data, not to be confused (or combined) as web design meets desktop in rich Internet applications.
[From Seek or Show: Two Design Paradigms for Lots of Data « Theresaneil’s Weblog]

The post describes two main groups of patterns, based on the way a user chooses what he/she wants to search (pictures linked from the article mentioned above):

The Seek Paradigm - The user has to input a search criteria in order to access some search results.
The Show Paradigm - The user is presented with several search results (or categories), and he may drill-down the search by clicking on the desired categories or selecting certain results to eliminate unwanted items.

This got me thinking in terms of offline work, and I can easily qualify both approaches as orthogonal to the online vs. offline decision. It is merely a user interface option, which is a good thing. To understand this, picture a lookup for book titles on a library. To go offline, one would have the client application to fetch them all from the server, thus making both kind of interface patterns possible without contacting the server. Similarly, if we wanted to know how many clients had already rented a book (assuming the algorithm implied a run through each client's details to look for their rental - I know, it's not the smartest way to do that, but it's only an illustrative example!), the client wouldn't be able to fetch all the other clients' details, hence not being able to use any of the search paradigms.

So the effort that has to be done, from the application logic design point of view, is related to how the services (those accessed by the user interface) access the information stored in the database. Let's draw some pseudo-code* for the book rental example above (the second example).

Approach 1: The all-powerful service. This is the kind of service we would have written up until a few months ago, considering any sort of offline execution to be an academic dream or an edgy unfeasible mostly unsuccessful experiment. Basically it assumes that it is being executed with super-user privileges, hence able to access all stored data at any time with no constraints whatsoever.
Aproach 2: The helpful, aware and conscious service. This is the service that knows that its code may be executed in many places, and so it tries to make a scarce usage of the stored data as much as it can.

Using approach 1 we could code a service like this:

(define-service get-number-of-book-rentals (book)
  (let ((all-clients (get-storage-data "clients"))
        (result 0))
    (dolist (client all-clients)
      (set result (+ result
                     (client-book-rentals client book))))
    result))

As you can see, while this would work perfectly on the server, the client would never be able to to the (get-storage-data "clients")call, given its lack of permission to fetch them all to the client-side storage. This is the sort of services that must be thought over and rewritten - or, better yet, automatically translated to something more useful!

By using approach 2, we would instead code something in the lines of:

(define-service get-number-of-book-rentals
  (:inputs (book)
   :storage-data (books-rent-by-all-clients (get-total-rentals-from-clients)))
  (let ((number-of-books 0))
    (dolist (client-rentals (filter book-rent-by-all-clients 
                                    :field "book" 
                                    :value book))
      (set number-of-books (+ number-of-books client-rentals)))))
    number-of-books))

Here it's clear that the service knows what data is going to be needed from the storage. With this information, the server can fetch it for the client when it wants to disconnect, and the client is now able to perform the same functionality without having to run through the clients (read, without having to have access to all clients' personal details). Relevant, noteworthy changes between the examples:

the service has to be declared with more information (like the storage-data requirements)
the storage data requirements code is always performed on the server. The client counterpart consists of a lookup on the client-side storage for the previously fetched information. This way there is no risk of information not being available.
If the service is to be run online, it also runs equally well, with no performance penalties.
There must be either a mentality change or a tool to translate services made with the approach 1 to the approach 2!

I'm already laying out some code to achieve something like this. Expect news in the following week or two.

*Disclaimer - despite being in a lisp pseudo-dialect, all examples could exist in the language of your choice. It's really just a matter of personal preference!

The search for functionality dependency

2008-01-10T00:40:00.001Z

Having had my PhD thesis proposal (previously presented here) approved by my university, it's time to move along to the next step: thorough description of the solution I propose, followed by its implementation.

The first important aspect I need to address is the grouping of functionalities that may be executed by a given user while offline. The ideal situation would be to pick up a specification of a Web application, then automatically produce a graph of available functionalities, with their dependency hierarchy. From here, the developer (or even the project engineer) could inspect what would be desirable to be performed offline, according to the requirements artifacts. Since we're also aiming for ease of use, on the produced tree, it would be nice to see the characteristics of each functionality, namely when related to access control permissions and data dependencies. This way we could gray out some of the available functionalities that could never be performed offline (e.g., we may want to force every functionality requiring a specific role to be accessed online only). By imposing restrictions, one could easily make a map of he needed functionalities. It seems almost to easy. What we're forgetting is how we're going to get the dependency graph. Yes, I intended to get those via a workflow graph. But let's face it, most Web applications are built without a prior workflow specification artifact ever seeing the light of day. Specifically, my case study doesn't have one (yet, however). So how shall I do this?

Write the aforementioned workflow specification by hand. This seems a lot of work to ask of the project development team, specially since all they want is some functionalities (they don't know exactly what or how many in a low-level, only the high-level use-case names are written in the requests, usually). It's also not possible to write it automatically, since it needs domain-specific information, usually more than what is already present (albeit very scattered) in the existing application. So this is a no-go.
Code walking. If we inspect each interface user action and the accessed data and services, we could obviously draw a graph. This however poses other problems. Are all programmers doing the right thing? It's not that uncommon to write services that check on other services without needing them, just to see some futile/temporary information, and forgetting about it later on. I know that doesn't happen in a perfect world, but "real" and "perfect" don't always pair along! But there is another show-stopping issue with this approach: we would have to start the "walk" somewhere, so we need to know all the starting functionality calls, along with the list of services itself.
User experience tracing. This is quite interesting in theory, if we ask users of several roles to perform their tasks, we can obviously trace the execution flow perfectly by reading the logs only. But we have no means to guarantee the users are going to access all the functionalities they are entitled to perform in our tracing period. That is only possible in a closed, controlled experience, by having users following scripts. But this kills the main purpose - those scripts are as hard to write as the workflow description itself, so it's of no use to us.
Functionality reification. By reengineering the application to have the functionality calls represented as objects, we can have the immediate dependencies graph automatically by scanning all user interfaces and listing the graphs of all calls triggered by buttons, links or other events (XHR based, for instance).

I'm going to put my efforts on this last one. There are other advantages with this approach, being studied by some colleagues of mine at INESC-ID and in the Fenix development team. The main ones are related to he presentation mechanism allowed - the organization of the Web application interface can be composed with such objects. I'll get into more details afterwards.

As usual, I appreciate any comments, tips or any other kind of experience sharing from you, dear reader!

Microsoft's Occasionally Connected Smart Clients

2007-10-27T12:04:00.000+01:00

There are two broad strategies for designing smart client communications: service-oriented and data-centric. When you have determined which of these to use, you need to make some fundamental design decisions to allow your smart clients to work offline. In most cases, the clients should be designed to use asynchronous communication and simple network interactions. Clients will need to cache data for use when offline, and you will need a method to handle data and business rule conflicts when the clients go back online.
[From Chapter 4 - Occasionally Connected Smart Clients]

This is a good overview for what I'm going to handle. Microsoft wants its PDA's, and other Smart Clients to be able to perform tasks while disconnected from the server. I want regular Web Applications to the the same, so I'm making the client browser just lie Microsoft's Smart Client. Let's overview their main points:

Approach: MS distinguishes a Data-centric and a Server-oriented approach to disconnected systems. The following graphic (taken from their article) illustrates that while data-centric approaches are not scalable due to their local replication of the entire database, the service-oriented approach also suffers from the lack of business logic (thus relying on the local cache without letting the server know it is not being contacted. We can envision server updates to this interface and/or business logic with painful consequences for the client... My approach is a mix between those two, for I want to design a client that is able to fetch a server module with both UI and business logic, and the necessary data for it to work. However, because we want the application to update the workflow, all client actions will still be postponed to the time that a server is reachable.
Conflict types: I've previously detected that all conflicts aren't the same. In fact, whenever someone edits a previously existing data (e.g., a student changing his email address), there is a simple data conflict, as the information on the client differs from the server's version. But this is a different case from an edition to a already deleted data (e.g., the same student tries to answer a question in a quiz that has already been removed by the teacher). MS article refers to the first case as Data Conflicts, and to the second as Business Rules Conflicts. Note that data conflicts may not be trivial to reconcile - consider a situation where two different entities change the same information, one offline, another online. Both want to change it, only one will succeed at first, but the offline client may need to merge its data, after having it sorted out with the other user.
Locking mechanism: while a pessimistic approach would minimize all sorts of conflicts, it is obviously not scalable to a widely shared data-set. So with an optimistic approach comes the need to synchronize and reconcile conflicts.
Reconciliation types: MS assumes the same three locations to do the reconciliation I've detected so far. If the conflict can be automatically merged, it will take place on the server. If not, it may take place on the client (with a simple user interaction dialog), or it may require the launch of another task that will contact other parties (e.g., two students posting the right answer to a question, needing the teacher input to opt for one of them).

The article further discusses CRUD operations as the ideal services to have in these situations (i.e., complex services would make client-side data synchronization harder and harder), and propose some strategies to handle task-based functionality dependencies.

While this is the closer I've read so far to what I want to do for my PhD, it's still lagged behind, as I propose a methodology to adapt existing applications (e.g., pick up existing web services and build from there), and I propose a conflict - and resolution mechanism - description that seamlessly integrates with the remainder application.

QUATIC07 - the humanistic version of the story

2007-09-27T20:19:00.000+01:00

On the last post I mentioned my findings from QUATIC07, and specifically from the input received from the software engineering doctoral consortium workshop. Now it's time to clear out what I learned from the whole experience.

Let's start this thing to mention this is my second event of this sort. I've attended to CAPSI 05, to present a paper entitled "Desduplicação sobre um conjunto de nomes próprios" (in english, "Duplicate removal in a set of first names"). The paper was presented by Helena Galhardas, that despite being the third author - my teacher of decision support systems at the time - she had both the recognition and the experience to make a proper presentation without the beginner's jitters...). So, despite having a published paper, I hadn't present any, so far. Enter the QUATIC event. I'm now faced with the opportunity to show my PhD expectations to other students and teachers of the same area. And present a poster to the entire academia/industrial audience on the conference's hallways. And have my thesis proposal published as a paper on a IEEE proceedings! I must say I was pretty excited about the whole deal!

Day one. I arrived at the conference to present the thesis to the doctoral workshop, SEDES. The conference was in portuguese, and the first interesting fact I noticed was the geographical tendencies of the participants: I was the only one who lived/studied/worked below the northern part of portugal. The rest were all from Coimbra, Aveiro, Oporto, so Lisbon (!) was a strange city among them :) But everyone was really looking for a chance to get feedback on their work, so there wasn't much time nor motivation for picking around that! Well, after receiving my reviews, I met personally one of the reviewers, and found out he is going to be part of my PhD jury. I had lunch with all of them, and proceeded to the remainder of the thesis discussions. And that's about it.

Day 2 and 3. The poster. I placed the poster on a wall in the hallway, and got a nice desk to sit at. This way, I got to show some supporting images to some enquiring passerby. I made an english 10 minute presentation about it, to the industrial world, and got some positive feedback on the subject (the technical questions were a bit far from what I was expecting; the academia is surely more prepared to criticize my work, as it is still a novelty to the others). On the last day I got to see a teaching perspective on software engineering. Miller's presentation about TSP and PSP were most enlightening and entertaining. The day ended with a dinner in a Brazilian restaurant with live music. I got to mingle a bit with people I had met but rarely had the chance to talk to. It was fun, but I have to admit I was pretty tired. I went to bed and slept for a good 7 hours, before waking up to finish another presentation (7 hours seemed a lot, as I had been sleeping from 3 to 5 each night until then!)

Last thoughts. I had a great time, the preparation was quite helpful in organizing my mind schemes, and the feedback was the cherry on the top of the cake. But now I need to evolve my work so that I can go to a IEEE well known conference - not that this wasn't a good one, but I need some heavyweight recognition when I'm facing a fulminating jury! Advice for the PhD newcomers? Don't miss out opportunities like this! It can be expensive, obviously, but there are organizations able to fund your research and your publications and presentations!

Going back to Emacs (is there any other TeX editor worth reverting to, when you're an emacsen?)

QUATIC07 - the academical side

2007-09-16T22:25:00.001+01:00

I've attended the last edition of the QUATIC conference, a three-day event on Universidade Nova de Lisboa, in Portugal. My main motivation was to participate on SEDES, a satellite workshop on Software Engineering PhD students. The purpose was to discuss ongoing PhD thesis and receive early feedback on them - while checking out other possibly related ideas and suggestions from the other participants.

I presented my PhD proposal, as it was by the end of July of the present year. It was a good experience, I received lots of appraisal for my chosen thematic, and also lots of critiques on the details of the thesis, my way of explaining things and my presentation. So lets start from the beginning, go straight to the end, then stop! :) (bored readers be warned, this is going to be a long post!)

My PhD thesis proposal is entitled "Offline execution in workflow-enabled Web applications". My goal started a bit broader, at a time where AJAX was starting to make its entrance in the development show-biz, I wanted to make desktop-like Web applications. Soon enough I realized I was foolish, because tens (hundreds?) of tools were immediately born to make Web Apps. look pretty, with lots of drag and drops, image overlaps, you name it. When something like Yahoo! releases their widgets, I can confidently stop trying to do something like that - I'm one student, they are one fully employed - read "payed!" - set of experienced developers. So I refined my goal and picked a real request - to make a Web Application run offline, without requiring external applications and without loosing any work. To do this I identified three main areas, or sub-problems:

Workflow decomposition into offline performable modules;

Offline client generation, and its functionality

Offline work reconciliation with the server

And this was what I had in mind at the time of the short paper submission. I was also invited to present a poster on how my work would affect the industry, with a short talk within the main QUATIC conference. That, off-course, set the stage for more interaction, as my poster was on a mandatory corridor to all participants ;)

Later on, I collected a set of information (from both research and opinions on the conference) that made my thesis improve itself. I'll try to present them here, as succinctly as I can:

Workflow decomposition is a relatively known problem, and people like van der Aalst have been working on Colored Petri-Nets (CPN) to work it out. The problem is now reduced to the question "How do I tell if I can perform this activity if I go offline now?". This is not as simple as it may seem, as I must ensure that there is a possible execution flow that won't require other entities online interaction. I also must ensure that I can bring all required data offline, to my laptop - and issues like access control or the amount of information arise here.

Offline persistency on a browser is now the purpose of tools such as Google Gears. Client-side generation using a high level language is also attacked by tools and frameworks like Morfik, XML11 or Google Web Toolkit. My problem is now to unify the specification of both online and offline tools into one. Later on I might also have to worry about technical details like client application transfer, shrinking, and a workflow-engine client clone, but not for a few months, for sure!

I was told the synchronization scheme I was looking for was already done by the nice guys from Redmond. The Occasionally Connected Systems framework is supposed to handle a local database (SQL Server Everywhere) and all the asynchronous messaging it takes to handle eventual internet failure, small period disconnects, etc, and that would include a nice and efficient merge of all offline data. While this may be true, there are a couple of issues that allow me to stay on the innovative side of the research: 1) The framework is intended to run on desktop-based applications. I assume that eventually Silverlight-based applications could be supported (as they grow the run-time virtual machine and bloat them with even more .NET libraries). But I want Web applications to do the same, and that's not been abridged, as far as I can tell. 2) The OCS framework is able to merge database information. But the types of conflicts I'm targeting are the high-level ones. I'm not interested in the last-edition time-stamp that is different. I'm looking for the product that cannot be bought because the store now is set to open from Monday to Thursday and I can only get there on Fridays. To make things clear, I'm not interested in a data merge algorithm, but in the elegant and simple way to integrate the specification of such kinds of conflicts - and their resolution strategies - within the application domain model and workflow definition.

This was a nice set of input. I was pleased. But I had some other things to fix, not so technical but about my approach to the whole PhD:

My sales-pitch - I need to fine-tune my goal statement into something that says "I want to describe a specific solution to a specific problem", and not "I want to solve al world's problems in two years, yay!!".

I also must get a supportive example. My example has fallen in the field of the non-problem - that was what I got by making it so simple I could explain it in a corridor walk... So the suggestion is to find a concrete requirement in a concrete (aka, real) system, describe the requirement, describe the problems in the current approach and conclude by suggesting how do we would want to do it.

I had to change a lot in what I was expecting to produce. Since I'm aiming for a development methodology (and the necessary framework or tools), I had to clearly state the inputs, outputs, all stages, the applicability/restrictions, the target audience, ad so on and so on. It's a shame there is no standards related to methodology description. I could certainly take advantage of one, now!

I'll post later about my personal experience on the conference - something from my human side :), and also to publish a copy of my thesis proposal. Here's how the poster looked like:

By the way, I have to say I consider myself fortunate, as this short paper is now published in a IEEE Proceedings! So, do you have any more advices, or experiences you'd like to share?

RIA and Flash

2007-04-20T13:54:00.001+01:00

Last Wednesday I went to a seminar on the application of Flash technologies to produce Rich Internet Applications. Adobe has certainly been busy for the past year, as we are far away from the "banners and intros" main purpose for Flash. Two main names pop-up for discussion: Flex 2 and Apollo. Flex addresses the issue of producing web applications with Flash and ECMA Script (the ActionScript variant), in the same manner one would develop a HTML+Javascript solution. The coding language is named MXML, and allows to mingle the UI and the client-side scripting. The main advantages are the Flash remote communication capabilities (either via web services, flash remoting, and other RPC alternatives) and the Flash multimedia embedding. It's quite easy to add a video playing in a given canvas! As far as coding artifacts are concerned, all we need is at least an .MXML file, like the following:

<?xml version="1.0" encoding="utf-8"?>

<mx:ApolloApplication xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical" cornerRadius="12" alpha="0.7" borderStyle="none">
    <mx:Script>
        <![CDATA[
            import mx.collections.ArrayCollection;
           
            [Bindable] public var people:ArrayCollection = new ArrayCollection(["one", "two"]);
           
            private function doAddClick(_name:String):void{
                people.addItem(_name);
            }
        ]]>
    </mx:Script>

    <mx:TextInput name="firstname" text="Type your name here"/>
    <mx:Button label="Add it!" click="doAddClick({firstname})"/>
    <mx:ComboBox dataProvider="{people}"></mx:ComboBox>
    <mx:FileSystemTree/>
   
</mx:ApolloApplication>

Yes, this is purely XML. One can add up pure actionscript files, wich, by the way, has some advantages over javascript (it's an OO language, for instance). The main drawback (yes, there is one) is the IDE is not free (unlike the compiler and SDK - those are FOSS! Also, there's no DTD, or XSD available to use in XML development editors, so we will probably have to stick to an Eclipse plugin (paid Flex Builder) until an open source alternative comes around. RIA has been forever tied to the sandbox concept: like all web applications, one can only access files that have been uploaded by the user in the present session. Another restraint is that a browser is needed to run the application. Here comes Adobe's Apollo. Take a Flex application, pass a short application XML descriptor, and with a couple of command line instructions you have a compiled and packed installer to a standalone windows/mac executable with the MXML contents. But they didn't stop there. Adobe also wants to catch up current HTML/JavaScript/AJAX developers, so Apollo also makes identical standalone applications out of pure HTML+JavaScript sites. But that's not all. to our coding pleasure, Apollo lets us access all Apollo's ActionScript facilities from Javascript, and vice-versa! This means one can code a javascript handler to access debug tracing, common dialogs, client-side file system, loke in the following example (taken from Mike Chamber's blog):

//called when button is pressed to select a file
function onFileClick()
{
 //this will trace out the string to the command line
 apollo.trace("hello");
 
 //get a reference to the desktop
 var f = apollo.flash.filesystem.File.desktopDirectory;
 
 //listen for the select event
 f.addEventListener(apollo.flash.events.Event.SELECT, onFileSelect);
 
 //open the browse dialog
 f.browse();

}

Again, the SDK is (yet in alpha state) FOSS, but the IDE consists in an extension to Flex Builder, and thus is paid. But the technology itself seems quite promising. See Adobe Digital Editions for a quite nice real-world example. In short, I'm impressed with Adobe Labs, and it seems there's more to come in the next couple of months!

Free Lisp Executables in Windows

2007-02-15T17:54:00.001Z

One of my latest projects has led me to think about making windows executables for a typical database application. The main requirement was using free software to make this project, so here's my starting point:

Clisp, 2.41 (2006-10-13) (full, with readline support)
ASDF
IDE: Emacs and SLIME (latest, from CVS at the date of this post)

The first issue I want to get behind is to make windows executables. If I can't make those, I might as well move to Python - but my programming language preference makes me stubborn about this one. I found out the magical command, ext:saveinitmem. But this alone wouldn't work, and I got to the conclusion SLIME was the problem - the binary image would try to output to slime, and result in an error. The fix is simple enough - just don't load swank, and make the binary without slime. I made a convenience build.bat file, with the following:

c:\dev\clisp\full\lisp.exe -B c:\dev\clisp\full -M c:\dev\clisp\full\lispinit.mem -ansi -norc -q -i build.lisp

(don't forget to change clisp's path to your configuration). One more thing I noted was the dll's clisp uses had to be in the same directory as the produced executable. (find them under "

Then, I put into build.lisp everything I want preloaded. ASDF, my project file loading, etc. and then, the magical function call (assuming an already loaded "test-app" package with a "main" function):

#+:clisp (ext:saveinitmem "test-app"

:init-function #'(lambda ()

(test-app::main))

:NORC t

:script t

:start-package :test-app

:executable t

:quiet t)

All fine so far. The executable is not small, but being about 5MB makes it smaller than the ~30MB I would get with the latest SBCL (1.01) for windows.

Next challenge, the database connection. This was a lot more troublesome than I initially thought. CLSQL was the elected choice, because I needed to access legacy data, initially from an Access .mdb file. I had to create an ODBC Data Source and use the clsql-odbc package to access it. I got this working in the clisp's repl, and then I made an image, run it and BOOM, all hell broke loose. Here is a thread where I got some valuable hints, specially from Ken Tilton, whose Celtk package I used. What happened is that clsql uses CFFI, with cffi-uffi-compat to interface to foreign functions defined in external dll's (in the windows case). Two issues ate more of my time than I would like:

Each defcfun (i.e., foreign function interface defined) must be evaluated after having used/loaded (only once) the foreign library. Ok, that makes sense. What puzzled me was that, unline Allegro CL, Clisp (and sbcl, while we're at it) requires us to repeat this use/load-foreign-library each time an image is (re)started. This makes it a bit messier, as I have to choose (1) to load clsql at the image execution-time or (2) fetch the needed libraries from the clsql project, and load them in my main (initial) function. I opted from the second approach, for image load-time's sake. The required lines are something like:

(cffi-uffi-compat:load-foreign-library "clsql_uffi" :force-load t)

(cffi-uffi-compat:load-foreign-library "odbc32" :force-load t).
CLSQL uses a variable to represent a null C pointer. this is ok, except that this pointer is created one time, so after clisp loads the created image, this pointer is invalid, yielding errors every time they are used. Solution? I've simply replaced all *null-ptr* with (make-null-pointer :byte) and *null-handle-ptr* with (make-null-pointer :void). This made the trick, although incurring in a small performancing penalty - I'm not that desperate, yet :).

After this issues, I was able to read and write to my access database from my (now 6.5MB) executable. On to the next issue, the GUI. Ahh, this made me think of lots of variables:

Language integration with the graphics (consider making calls for foreign functions vs. using simple abstractions in lisp);
Portability, I wouldn't like to build different GUI's for Mac and Windows, for instance;
Prettiness. Ok, It may be the best toolkit ever, but I may like to have, say, the xp look and feel. Or some other...
GUI design. There are some nice GUI designers out there, and there are some nice lisp abstrations to make the same result. What to choose?

Given this pre-requirements, my options for free were wxCL, graphic-forms, celtk, Ltk, and cells-gtk. (This list may not be comprehensive, feel free to comment, tell me about the amazing />

I tried wxCL first, as I'd already used wxPython, and felt at ease with wxGlade. So I started using it, created my first hello-world widget, and bang, had a whole lot of troubles, cffi related and others. I guess that they could be handled, but I would still be attached to the wxWidgets theme, not very handsome to me. For the same reason I ditched cells-gtk... I moved along, on to Graphic-forms. This is still in alpha stage, but is looking nice! Although, the interface is only for windows forms, and that makes it pretty useless in a mac/linux box. On I went, to the tcl/tk world. At first I thought I'd be attached to a specific theme/motif, but I've learned about Tile, and this made the trick for me. So, how do I choose between Ltk and celtk? Well, Ken Tilton's Cells project, basically, prodded me towards celtk :) It's just a programming language decision, cells can be used to bind behaviors to certain parts of the GUI in an oh-so-elegant way, I was hooked soon enough!

Ok, so what's the downside, you may ask? Celtk is still a work in progress, and is kind of not-ready for quick use/deployment. I also had cffi-related problems, simillar to those I had with clsql, and I worked around them with the following addition to may main function:

(cffi::use-foreign-library "/dev/Tcl/bin/tcl85.dll")

(cffi::use-foreign-library "/dev/Tcl/bin/tk85.dll")

(cffi::use-foreign-library "/dev/Tcl/lib/tile0.7.8/tile078.dll")

There's also lots of absolute paths to fix, in a decentralized way, the asdf is not up-to-date (Ken relies on ACL's *.lpr files), there's no readme/howto/start-here, and basically no homepage with this information. I think Ken is trying to get the project a home in http://common-lisp.net/, and meanwhile the comp.lang.lisp newsgroup will do to discuss issues we may have.

What I can say is that I've put his demo project, along with a sample MSAccess data access, in an ~8MB executable. I'm sure I can optimize this, but for me it's fair enough, and it loads fast enough for me!

Hope some of you may learn something from this. Feel free to shoot any questions, or suggestions you may have!

On thread persistence and migration

2006-11-28T19:12:00.001Z

Having contacted the Hop authors, I got a good feeling on their progresses, and I'm looking forward for the next (beta?) version. Also, they suggested me a PhD thesis, "A Portable Mechanism for Thread Persistence and Migration", whose details can be found in Wei Tao's Ph.D. Research Page.

Her dissertation is a good explanation on how a Java program may be adapted into having support for thread suspension and resume. She based her work on a complete code re-writing (both at source and byte-code level), following a (theoretically correct) set of transformations. In short, a program must emulate the creation of a continuation on each code sentence, using an exception mechanism to capture the state of the running program. This exception usage is handy, because after catching an exception the code may throw it again, or store (persistently) the execution state. Obviously there's the problem of non serializable resources having to be saved and restored, and the local, temporary variables. Tao, cleverly leaves this issue to the application creator, with two resolution strategies:

Creating a wrapper class to the resource, that knows how to close and recreate it (e.g., close a file port and re-opening it a few days later);
Setting the an existing object field to the resource placed in a local variable;

It's relevant to note that Tao relies on the use of the transient modifier, that makes such a field null when the object is serialized - having to be re-created on deserialization.

The process is heavily focused on the Java Virtual Machine, and the transformation is not proved to make such sense in other programming languages/environments. Nevertheless, the byte-code transformation strategy is well documented and several performance considerations are made throughout the dissertation. I, however, believe that this approach may be simplified - particularly the exception-based technique used to save/restore execution state - when the application is written in some other languages, like Scheme, that natively support first-class continuations.

Her motivation to this technique is well founded, focusing both portability (she may employ this transformations on several applications, and even with other programming languages, as long as their core is closely related to Java's) and code mobility (inevitably the best reason to save thread execution state and to be able to restore/resume it later, although "Moving long running applications from desktops to laptops and back also has clear appeal").

One interesting point she states from the beginning of the report is that making a thread persistently saved is quite similar to having it migrated to another place, the difference being that the store is the run-time environment of another process/machine.

If you're interested in execution-flow migration, thread state, and similar issues, I advise you to read this report, or at least to skim through it.

Mosquito Lisp

2006-11-20T20:17:00.001Z

I stumbled upon Mosquito Lisp, and thought for a minute that it could be the adaptation to Termite's concepts into a more web-oriented strategy. But I was wrong, as it is only more focused on security measures, and leaves the main attraction in Termite behind - not only it does not allow the exchange of continuations, as it doesn't even consider them in the language (as with macros, now that we're at it!). This can make the dialect extremely simple to grasp and employ on real world cases, but it fails to make academic experiments with abstractions and other language tricks possible (at least throughout several machines!). Here's the link to the introduction:

Ephemeral Security - Introduction to Mosquito Lisp

Application interaction models

2006-11-18T11:56:00.001Z

I have assembled the interaction types I want to make possible in the near future. They assume a similar architecture to the one verified in current Web Applications, that is, a client-server architecture, a standard (mainstream) browser running on the client and an application container of your choice running on the server. But, unlike today's regular usage of this architecture, I want to allow some other application interaction models:

Client-Server code mobility, where the server may detect it is unable to compute every requested operation, and sends some of the (possibly still running) computations to the client for it to finish them. As I depict in the preceding diagram, after the client knows the code to the computation he needs to finish, it can gain the ability to perform it without accessing the server (unless there is some server-side information that needs to be fetched, obviously).
Disconnected Execution. This sort of interaction is interesting as the client is able to work while being offline (or at least, disconnected from the server). The only requirement is that all of the functionality (perhaps organized in functional modules) and the related domain data must be fetched from the server: As one can see, this may pose other challenges, such as domain data access control (e.g., I may not be able to have full control to some objects that were being used by the server to produce the interface, and thus I cannot have that (whole) object on the client side).
Server replication, to grant reliability in the provided application. While there are several approaches to do server replication, all of them must allow messages that arrive to a front-of-the-line server to be replicated to all others (or at least the effects of processing the received message). This can be done by the exchange of continuations (see the Termite approach, in my previous posts).
Inter-client interactions are the most uncommon model, when compared to existing uses of Web Applications. In the "several interactions" arrows can be two main types of communication. There can be only data exchange, like in instant messaging, white-boards or on-line games. But there may also be execution flow exchange. You can have two clients making parallel computations, in a sort of collaborative processing. You can also have ubiquitous execution, like having a smart-phone "pick up" the application execution from the desktop computer's browser and carrying on with it.

While this list may not be complete, it is my starting point. Off course I'll add more examples or complete/modify the existing ones if I see it necessary.

Have a nice weekend!

Mobile computations example

2006-11-15T01:42:00.001Z

In an attempt to use a new Web diagramming tool (Gliffy), I just put into a simple time-line an example of what I want to make possible with my project:

Basically it is a client that starts by asking the server for a specific code execution. the server, however, opts to stop the execution midway, and transfers its work in progress to the client. Now a couple of noteworthy behaviors can be verified on the client:

The client finishes the remainder of the computation, not contacting the server anymore in the process;
The client now knows how to make that kind of computations, so it will never bug the server again for that.

I'll welcome any comment on this idea, specially constructive criticism :)

How distributed mobile computing resembles a JIT compiler

2006-11-14T15:43:00.001Z

The Just In Time (JIT) compiler used in the Java Virtual Machines (JVM) is the tool responsible to compile Java bytecode into native platform code, thus granting the application improved execution speed on certain operations.

How can this be comparable to distributed mobile computing? Like I stated in previous posts, I intend to be able to migrate code that is currently being executed into other machines, and resume it's execution there. Let's analyze what this means from the client application's point of view:

The client is viewing a page that accesses a procedure that is currently being executed on the server. Thus it calls it with a call (possibly an asynchronous one) to the server, and sets a callback function to receive the server answer;
The server opts to send that computation to the client, so it does so (let's assume for now that it uses a Termite-like approach to do so);
The client has now the task of storing the received computation is some JavaScript callable form, so that it can call it directly instead of using the RPC and callback he used before.

To do this, he changes the way he calls a given functionality. Which is pretty much the same thing the JIT compiler does. The JVM stores a couple of Virtual Tables (V-Tables) with the addresses of every functions. In the first, it stores pointers to the source code, while the second stores pointers to byte-compiled functions. The first times the code is called, the JVM uses the first V-Table, but calls the JIT to make the compilation and fill in the address in the second V-Table. This way, subsequent calls to the functions will be directed to the byte-compiled file, directly, in a transparent way for the application code.

This can be used as an inspiration for code mobility: Both the client and the server may store tables for each procedure that may be subject to migration, associated with the local address or the particular remote call mechanism, along with a flag that states which method is used currently. The problem still is what information is relevant for each function execution. This single table would suffice if we wanted static code migration. But if we want to migrate running code, we have to consider every function that can be used within the running code.

This topic is still fresh, and I will have to do some experimentations before I can have a more conclusive opinion. But I have a good feeling about this mechanism when associated with a Termite-like code migration technique!

Inria's Hop

2006-11-13T20:20:00.001Z

I finally got to read both technical papers on Hop (found in Hop Home page), and I can now make a reasonable comparison between their approach and Termite's.

First let's describe Hop's in a few sentences. Being aimed at Web applications development, Hop is mainly a programming language that separates the application logic from the interface definition, while maintaining a single coding artifact for the application. The language is built upon Bigloo Scheme, so that servers may import several useful libraries. The core features of Hop are:

server services - defined just like a regular function, with the macro 'define-service', can do basically anything on the server.
interface - created with macros that resemble the HTML tags, interfaces are a declaratively constructed HTML tree. They can contain scripts with Scheme code of two different natures:

Client-side code, translated automatically from to JavaScript;
Server-side code, mixed within client-side scripts but prefixed with a special character. This code is translated to the appropriate asynchronous server calls, and every exchanged object is automatically mangled/de-mangled (JSON can be used to read server responses on the client).

One of the things that pops out when you see a Hop application is it's elegance, as everything is written in a single file using the same programming language, despite addressing the two important application "strata" (server logic and user interface). The continuous bidirectional communication (and data exchange) between the client and servers is accomplished via asynchronous remote function calls to the server and by setting events on the client that are triggered upon a server's reply - enabling both push and pull communication models.

Another interesting fact is the execution model Hop uses. It is based on several engines running for a single application: One for the UI processing, and other(s) to the application logic. The URL start-point loads the UI engine, and henceforth it is responsible for calling other engines as other functionality is needed.

But Hop isn't perfect. Let's take a closer look at the service model. As it is, Hop creates an URL for each server call needed in the UI strata. Each URL is stored in a server-side table, that is never cleansed - even temporary URL's used for anonymous services are kept indefinitely. This could be resolved (as the authors briefly point out) by exchanging closures (both the server function identifier and all the necessary variables in the environment) with the client.

By now, we can understand another weakness of Hop. Hop uses SCM2JS, which translates Scheme code into JavaScript and allows both languages to work seamlessly within the same memory space. However, as technically close as the languages may be, the translation fails in some key aspects, such as the Scheme support for continuations. The authors state that this could be done whether by the use of trampolines or by the translation into Continuation Passing Style (as specified by Henry Baker's paper), but would always create performance penalties. However, with support for continuations, Hop could easily get all Termite's mobile code functionality, like replication or execution migration between processes.

In my opinion, even if Hop can do such code migration with continuations, it's still weaker than Termite, as it's goals are (currently) too low. They want to build simple Web applications based on interactive operations. They wouldn't be able to build a peer-to-peer application with a Web Interface using code migration between clients, as the concept of client is mangled in the Scheme code as being the UI strata. Termite's approach, on the other hand, clearly considers processes, and that seems to make sense to me, as both clients and servers can currently play the same roles, with bidirectional client-server communication.

Before I wrap-up this Hop analysis, it's relevant to point out other weaknesses of the SCM2JS:

exact Scheme numbers are currently mapped to floating-point JavaScript numbers;
Errors in Scheme are not necessarily errors in JavaScript, as the translation leaves erroneous Scheme code in executable JavaScript code (like the concatenation of two objects, (+ "a" 2) is translated to "a"+2, which yields "a2" and not an error);
features like call/cc and eval constructs are not translatable, yet.

I hope Hop evolves into a great tool. Also, Bigloo Scheme, as well as Hop, need to be executable on Windows boxes, to gain the deserved popularity!

Disconnected browsing

2006-11-08T02:48:00.001Z

One of my current activities' main goals is to produce the necessary infrastructure for a web application to be used offline. While the term disconnected browsing is becoming obsolete - Web applications now offer users a decent graphical (web) interface, not a bunch of HTML documents interlinked in a browseable manner - I'll stick with this denomination for the time being, until a better one comes arround. My research for now has been centered on the first Web architecture, the browseable type. Even so, up until 1997 there's not much public research on this matter, and the material I've found isn't what I'm looking for. In Web Intelligent Query - Disconnected Web Browsing using Cooperative Techniques - Kavasseri, Keating, Wittman, Joshi, Weerawarana (ResearchIndex) - it's clear that even then the Web model wasn't appropriate for disconnected browsing. They identified that, in order to browse the Internet, one would have to:

Know a set of known starting points;
Have a constant network connection;
Filter lots of useless information;
Put up with bandwidth shortage and frequent link faillures (mainly with mobile devices, but also by elected network disruption, e.g. for battery power saving).

The authors state right from the beggining that the proposed solution targets static Web information. I know, this makes this solution useless for today's Web model, but I wanted to read on, just to check if there was some valuable idea. But they propose an architecture based on an "intelligent" proxy, one that stands between the browser and the Web, and is responsible for making both IR (Information Retrieval), based on users requests and IG (Information Gathering), based on guessed pro-actively retrieval of information (Cooperative Information Gathering: A Distributed Problem Solving Approach - Oates, Prasad, Lesser (ResearchIndex)). They have some interesting related work, but the one that caught my eye was a paper on usage patterns detection, for production of browsing related tasks automations (Integrating Bottom-Up and Top-Down Analysis For Intelligent Hypertext - James, Margaret, Recker (ResearchIndex)). The concept of disconnected browsing for them demands a very strict flow, from the user (client) standpoint:

The client requests a set of information;
The proxy makes its magic, fetches data from the web - it is permanently online! - and processes the request;
The client asks for the fetched results;
The client processes the information;
The client sends feedback to the proxy.

During this steps, the authors consider that being offline during steps 2 and 4 grants the client the designation of a disconnected browser... I intend to shed some light of the concept of disconnected execution of Web applications. This has lots of non-trivial obstacles, like

the existence of dynamic information;
the concurrent access to mutable, persistent server-side information;
the nature of the graphical interface being no longer a single (nor static) HTML page.

The usage model is also diffente, as the client must pre-fetch some portion of application and data, and then must be able to go offline and play arround with it. After he comes up online, the application must somehow synchronize his changed information with the server's - which, by the way, may envolve complex conflict resolution and/or merge rules for specific information types. From this paper, however, there are some good contributions to my project:

Disconnected information becomes much smaller if we know a priori what to download. So it would be a good idea to detect what the user commonly does, his usage patterns!
It may be usefull to have a separate agent responsible to fetch the application code/data to be downloaded to the client. As this can be computationally heavy task, having the web server do it may cause serious performance issues.

Right now I'll keep looking for other disconnected Web usage papers and projects - I'd like to see something recent, at least not older than 5 years! I know that when I reach Coda's documentation, I'll be too far from the model I want, but eventually I'll have to get down to it, as it has some good, well documented merging techniques.

Concurrency in a web platform with distributed computing

2006-11-05T14:42:00.001Z

Right now, it's clear whatever platform is to be the key for a Web application success, it must be based on a client-server architecture, something with the look and feel of the so-called Web 2.0 applications. That is, I want to use a Web platform to implement the aplication GUI, just as if it was a desktop application.

Wrt. the concurrency model, I would like to have multiple clients accessing simultaneously to the same application. Off course faillures on certain nodes (either clients or servers) must also be considered. Lastly, I want to be able to make computations on both the client and the server side.

Without having a Web architecture, I can find three approaches to the concurrency aspect:

Shared space: By using a memory space shared by all interested nodes, one can make applications by building (complex) contraptions with locks and semaphores. It's tedious, error-prone, and is generaly considered to be avoided in medium-large applications.
Message passing: Using the Actors approach in which everything is an object that is operated solely by exchanging messages with it. Most of the times this is simplified to the model where only processes (or nodes, or threads) send messages to each other (like Erlang)
Declarative data flow: this is the model used by Termite, and I've described it extensively in the previous post. In short each process can send messages, data, and execution-flow to another process.

From these three, the third seems to solve more issues:

Doesn't need the coding complexity needed in the shared data management;
Does not demand separate programming of paralel processes;
Declarative DSL's save lots of code lines, with abstractions based on continuations replication between nodes

With the Web architecture in mind, we must consider at least the following questions:

How are computations going to be transferred to a given client or back to the server?
How will the computation be transferred at any point of the application execution?
Given the bandwidth current limitations, how are computations going to be serializable? In processes, continuations, closures? What is the relevant minimal data set that need to be passed along the wire?
It is a fact that the client must have an engine (in ECMA Script, I suppose) to process some part of the application - either the user interface or the domain application logic. This engine must be prepared to receive this sort of computations from other nodes (as well as to send some back to them).

This questions are good guides to the next stage of my research. I'll make sure to post the answers I (hopefully) find.

Termite - the light at the end of the tunnel?

2006-11-05T01:33:00.001Z

Last month I started looking at Termite, a Scheme-like language to produce concurrent and parallel programs. It's now time to make the relevant comments, based on the paper (read it here) and directed to my PhD purpose - Webapps Interfaces development in the Web 2.0 new world.

The first thing worth making clear is the concept of distributed computations. These are concurrent programs (lightweight processes), potentially physically isolated. The data transfer between each process is made via message passing. This is based on the Actors approach, and can also be verified in Erlang. This model carries the burden of the need of functionality on the programs code. In other words, mutation cannot be done in Termite. Fortunatelly, it is possible to emulate the same behaviour by representing the "mutable" state in a process, and using a message dispatch mechanism (and representing an object in a separate process).

Processes are isolated, even if they are on the same physical node. This makes the concurrent model quite simple, as there is no shared memory, no mutexes to handle and, additionally, there is no need of a distributed garbage collector!

The message sending is done using a single mailbox per process, with assynchronous sending (the send and pray model!) and synchronous receiving. Also, Termite assumes that connections cannot be trusted as being reliable, thus assuming the possibility of failure. Since there is no message delivery guarantees, there also are no correct order of reception guarantees. Termite leaves the burden of dealing with this to the application code, by using timeouts mechanisms. Luckilly, Termite has a way of propagating exceptions between two related processes, using Erlang-like bi-directional links, but also using a one-way-only link between two processes. Finally, Termite supports tags in each sent messsage, thus being able to mimic a multiple-channel model in a single-channel architecture. This tag system can also be used in the timeout message detection system!

One interesting aspect is the worarround to make every object serializable. Non-serializable objects are proxied by using a process to represent them, so e.g., each port can be serialized using only a PID! This means we can send a message to the screen, or receive a message from an input device, which gives us a great sense of liberty on our programs!

When comparing to Erlang, aside from the Lisp-like syntax and the one-way links between processes, Termite also provides support for first class serializable continuations.

The related work also indicates that Kali uses a lazy closure code transmition between processes, along with a cache system, that could help improve the code!

After reading this, there are a couple of questions to ask about Termite: What does it take to build a web framework using this kind of comunicating processes, and what's missing to be able to fulfill my PhD requirements.

The first question is answered in the Termite paper, by Dynamite. Unfortunatelly, I wasn't able to find any proper documentarion/web page/binaries/other material related to it.

So let's talk about what Termite can do for me, and what it can't do. If we want to use the comunicating processes model, we can envision a client being a process that sends messages to a given node on the server. This node can be connected to others, making server replication a trivial task to do (the Termite paper illustrates this behaviour). I would like to have reliability on the message sending, so this would have to be added as a layer to the Termite framework, using some sort of timeout detection. I would also like to know that the delivery order is respected in every message, so that should also be granted at the message-sending protocol level, as well. Let's say that this would be enough, for now (and leave "minor" details like scalability and speed behind!). Since we can propagate messages between Termite processes, the trivial way to achieve this would be to integrate a Termite engine within a JavaScript client-engine. I can think of a couple of ways to do this. The first is simply to implement the Termite (and all of the Gambit-C dependencies) in Javascript. The other is to analyse the messages that need to be sent through processes, and produce a platform independent protocol (at least, to enable a JavaScript client to talk to a Termite one :) )

I know this is still a bit hard to achieve, and I should look into other approaches before trying this. But just imagine being able for a server to start by making all of the application computation. Then, as the server machine decides it is fed-up, or if it is simply bloated, then it spawns another server in a neighbour node, and transfers the computation to it. Also imagine a more complex load-balancing mechanism being implemented between the servers. It's all very interesting (I guess this is what should be being done in the Dynamite project!), but I must confess the fun part is between clients and servers. The server finds out, after a while, that the client is able to do some processing, and it transfers a continuation with the job to be done on the client-side. Since each process can update the running code during its execution, this would be easily achieved. Now the sky is the limit, since not only we can have code execution being swaped from the client to and from the server, but we also can envision collaboration between clients, without any kind of server interaction! If this isn't enough, lets remember that the communication is reliable. Oh, and that by now we already have a fault-tolerant (replicated) server system! We could even imagine a client being able to detach itself from its server, going offline, keep on working and then synchronizing with any server when it gets back online. The implications on these kind of scenarious are a bit different than the message passing model, as we need to consider the domain structure of the application, access control, the ammount of data the client needs to own in its machine to operate, among other important issues.

Sorry for the long post. I guess If you got this far, you might as well have some opinion on this matter, so please, leave a comment!

Ta ta...

Lisp Interpreter in JavaScript

2006-10-22T01:11:00.001+01:00

After checking out this JavaScript file, and another similar approaches, I found out that this is not exactly what I was looking for. Here's what I have in mind, for a virtual machine on the client-side:

I need it to run code specifically designed to it. (Right now, anything must do, but I need it to be able to translate it to valid client actions - aka, use something like JavaScript, DOM, CSS, etc, etc.
I need it to be modified from the server (i.e., to upload code changes)
I need to have some sort of support for a set of language constructs and basic types, including continuations. So I probably need to serialize everything.
I need to consider the drawbacks of maintaining all of the variables in the continuation's environment. One simple closure may be small, but the full environment - needed to make good use of dynamic variables - can become quite large. Note that some tests are in order to make this assert valid!

The downside of the Lisp Interpreter I checked is that everything is automatically translated to a Javascript object. That's not what I need, since I need the code to be able to travel back into the server. I only need it to be evaluated on JavaScript. (This also means I can use a portion of this code, like the tokenizer and/or parser!)

More to come...

The Law of Notepad

2006-10-21T12:24:00.001+01:00

Taken from Continuing Intermittent Incoherency blog:

The Law of Notepad: if creative people can’t make something awesome using the lamest of production tools, your platform is gonna loose.

OpenAjax Hub

2006-10-19T21:21:00.001+01:00

Here is a quote from Coach's blog, related to the main challanges that OpenAjax is trying to address to get interoperability:

coachwei.com :: OpenAjax Update - Addressing Key Challenges for Ajax Adoption :

# Toolkit “loading and unloading”(the capability to load and unload more than one Ajax toolkits into one page): For example, right now a lot of toolkits overload the “onload” function, which will certainly cause collision with another toolkit;

# Name collision (the capability to deal with variable and function name collision). For example, quite a few Ajax toolkits today use “$” as a function name. When they are all loaded onto the same page, what happens?

# Event Hub (The capability to have interaction between different Ajax toolkits). The key requirement here is to enable some kind of “event” propagation so that an event generated from toolkit A can be processed by toolkit B. An example is that the user drags an item from Toolkit A and drops it on Toolkit B – how should this two toolkits communicate the information related to this event? The proper solution is to provide a common event hub that each Ajax toolkit can publish and subscribe events. Initially, we do not have to define the details of different event objects, which can be defined in the future.

# Markup scanning (the capability for an Ajax toolkit to scan an HTML document to pick up the interested markup segments for processing): many Ajax toolkits allow the usage of markup to define components within an HTML page but they all have their own mechanism of scanning the document. If more than one toolkit is loaded, the HTML document maybe scanned multiple times with the danger of causing significant inefficiency and collision. Providing a common mechanism to scan an HTML document would solve this problem.

# Network Hub (server connectivity management): most browsers allow only limited number of connections to the server (for example, Internet Explorer allows only two). If each Ajax toolkit uses its own mechanism to handle server connectivity, it is highly likely collision will happen if more than one Ajax toolkits are loaded on the same page. A proper solution is to provide a common mechanism that centrally manages server connectivity and serves all registered toolkits.;

Apart from this list, they point out other missing factors, but that shall not be addressed by OpenAJAX. They are performance issues, the multiple toolkit bloating problem, whether there are security issues with AJAX, and if so, how do they die. Other interesting problems arise with mobility and offline execution of AJAX-based code, as well as with application development and maintenance.

These last two are particularly interesting to my work. The mentioned post talks about a blog with mobility issues: SitePen Blog » mobile.

HTTP is dead. All hail Web Messaging Protocols!

2006-10-19T02:45:00.001+01:00

It has become very clear that HTTP doesn't serve current Web needs and purposes. So why does it remains the sole winner of the communication protocol in the Web? This is simple, really. Every browser knows how to speak HTTP, and since it's a pretty simple protocol, nobody seems eager to move a multitude of users into a paradigm shift.

The tough question is related to the solution, instead of being centered on the problem space. Where do we go from HTTP? What do we need? Most aplpications today are struggling to find a proper way to communicate with the server. Comet strategies provide bi-directional messaging between a client and a server. Internet Messaging Bus goes a step further, granting the following properties to the communication protocol:

Guaranteed message delivery
Guaranteed order of delivery
Once and only once delivery
Both client pull and server push data fetching models.

This seems to be the end of the AJAX hype, as it is know. That is, the XMLHttpRequest, by itself, doesn't grant all those nice things to any WebApp. However, the word seems to be catchy, and people seem to want to stick with it for a bit longer, and the current bet is a messaging protocol.

See:

Process execution mode swapping

2006-10-18T20:09:00.001+01:00

Another aspect that is relevant for my PhD is the exchange of information between the client(s) and the server(s). To make a pictorical description, imagine a situation where a client walks into a web application, which is running entirely on the server. The server then sends some of the computations to be executed on the client, e.g., for performance reasons. After a while, the client may take too long to execute a certain computation and transfers it, along with the already processed information, back to the server.

This situation can be compared to the Java Bytecode execution done by the Just In time compiler. The VM starts by interpreting code, then after a few iterations, it boosts up the compiler and the code that was being called is now fetched from a compiled source.

What I mean to inspect is if I can take some of these ideas and port them into a Web architecture.

In SISC - Documentation, we can see that the Scheme interpreter can also run byte-compiled Java code. I must now find out how does it swap the execution context between those two modes, and what considerations are being done.

Interpreters in JavaScript

2006-10-18T19:23:00.001+01:00

Before discussing Termite (it deserves a post for itself), I'll post a series of links to attempts to write language interpreters in JavaScript.

First, the motivation: As I need to create a framework to build web applications with certain characteristics built-in the client-server communication infrastructure, I need to do constant experimentations with the client capabilities. Considering that the majority of clients are able to run a javascript interpreter within the browser, it seems to be the best bet, for compatibility sakes.

A list of links to different approaches follows. For some pages, The only interesting reading material is on the Javascript that is embbeded in the source html code, so you have to dig into that.

Interactive Scheme (see http://fresh.homeunix.net/~luke/jscm/scm.js.txt)
Lisp Interpreter in JavaScript
A Brainfuck interpreter in JavaScript
The Little JavaScripter and the Little Scheme
Joe Ganley's Lisp in JavaScript

Conclusions on whether this will or will not be usefull may come up on future posts!

A Common Lisp Post

2006-10-15T13:03:00.001+01:00

This is my first post sent via my Common Lisp Atom interface. It uses the JLinker to call the Java Google client API methods.

Termite - concurrent and parallel programming in Lisp

2006-10-10T13:39:00.001+01:00

This is a simple tutorial for Termite, made by its author:

Bill Clementson's Blog: Update on Termite (A Lisp for Concurrent/Parallel Programming)

After reading the paper on it I'll make a proper review, but I like the side remark on a file upload procedure definition:

this program has about the same number of lines of code as a minimal "hello world" program in Java.

Google Web Toolkit experiences

2006-09-26T10:58:00.000+01:00

I just started toying arround with GWT (Google Web Toolkit). Here's my thoughts so far:

[GOOD THING] It makes the building and running on the development environment simple. It doesn't offer the flexibility of a running interpreter, but it provides two commands (scripts), one for compiling and another for running a box (development shell) that is an application container (tomcat based) with a small browser for a quick test.
[GOOD THING] An application has to import only a couple of jar's to run
[BAD THING] We have the old Template Model problem. We have two source folder structures, one "client" with the java files, and a "public" with the html templates, images, CSS, and other files to be published directly. The HTML templates should contain id's called "slot1", "slot2", and so on, for each object that is placed with Java to be attached to a certain place in the template... Is this horrible or what? The good thing is that google also lets us do (almost) everything without a template, creating defaults!
[BAD THING] I just noticed something weird in the demo application, the KitchenSink. a page with a floating dialog box, that warns with the following:

"If you're wondering why there's a list box at the bottom, it's to demonstrate that you can drag the dialog box over it. This is noteworthy because some browsers render lists and combos in a funky way that, if GWT didn't do some magic for you, would normally cause the dialog box to appear to hover underneath the list box. Fortunately, you don't have to worry about it -- just use the GWT DialogBox class." Now this is (1) weird, as it seems we're still dealing with black-magic in order for things to appear right on the screen, and (2) wrong, as the google shell browser still shows the listbox over the dialog box, if we drag it towards the right (oddly enough, a left drag runs ok...)

After having some meetings about the future of my ...

2006-09-21T15:39:00.000+01:00

After having some meetings about the future of my research in the ESW group (at INESC-ID), I've set a few goals to my PhD. I'm now a part of a project that will integrate a workflow management system into a Web Application development methodology. My contribution is centered on the following RIA features:

Easy rich content development on a web application
Disconnected browsing, used to enable specific parts of the workflow to be worked on without requiring a permanent Internet link.

The second point is my main goal, as the first is being more and more common every day (do a google search for 'AJAX' and check it out!). To achieve disconnected browsing, one of the biggest challanges is to find out what is going to be the persistency model - given that the information is always shared and interacted with between two possibly remote machines.

Some questions to be answered:

What is the minimum ammount of information needed in the client for it to be able to execute operations offline?
What are the security issued underlying such an information sharing?
How can workflow activities be moduled so that they can be run isolated from the rest?

Here is a summary of the activities I fullfilled i...

2006-09-21T15:17:00.000+01:00

Here is a summary of the activities I fullfilled in my PhD, during this year:

Curricular Courses: I have completed the curricular component of my PhD, with the following courses:
- Formal Software Development: Based on the B-Method, this course was about the development methodologies with the help of formal specifications;
- Advanced Topics: I studied the Desktop applications interfaces development tools, their types and characteristics. I also read Jenifer Tidwell's "Designing Interfaces" book.
Research:
- I made a study on remote desktop interactions, where I calculated the required ammount of network bandwidth to enable common user interactions to be executed in remote applications, such as mouse moves and clicks, or key presses. The conclusions were satisfatory, as today it is possible to enable complex interactions over the web, even if they are mapped as a set of basic - and non-optimized - operations.
- I analised several tools that help the development of Rich Internet Applications (or RIAs).

AJAX and Accessibility - Standards-schmandards

2005-12-13T17:03:00.000Z

I read an article from March the 1st of this year, on AJAX and usability. It's a nice, simple, straight article, easy readable.It also has a few references to other articles explaining blogs (as well as a long comment list!). Here's the main points of the author on usability: "Improving accessibility of AJAX forms Fortunately there are some things we can do to increase accessbility of AJAX forms. Here are my recommendations: 1. Inform the user at the top of the form that it requires javascript or detect javascript automatically and warn the user when it isn't available. If the form has many fields you will spare your users a lot of frustration. Everyone hates filling out a form just to find out that they can't submit it. 2. Inform the user that the page is updated dynamically. This is especially important for screen reader users and will help them decide when to trigger a re-read of the page. 3. Make it possible to recieve an alert when information was updated. This may not be practically possible depending on the complexity of your form but will help a screen reader user a lot. Alert boxes are read by the screen reader and are usually displayed together with a sound. The checkbox should be displayed so it is clear that it is not part of the original form. 4. Highlight recently updated areas for a short period of time. This will help sighted users understand what just happened. The nice folks over at 37signals have dubbed this 'The Yellow Fade Technique' but you can use any colour you like. Check out Adam Michela's code for another way of providing the fade."

Morfik - the Javascript synthetizer

2005-11-25T16:23:00.000Z

I've recently found an interesting, promising work, another framework for Web Applications development. So what's different in this one?

Taking Morfik's CTO's words, ?Morfik has developed a unique JavaScript Synthesis Technology (?JST?) that allows developers to use a powerful integrated design environment together with a high-level language of their choice to create reliable applications comprised purely of HTML and JavaScript?.

Since the framework isn't open-source (so far, at least) all we can checkout is the two Flash demos on using the program to develop basic applications, and a few white papers and communications they release to the public. From these, I'm able to comment on both their mission and the program itself.

Morfik's main goal is to take jobs away for "rocket-scientists" who want to develop software for well-paying companies. So far, the so-called AJAX applications have been hard to implement, given both client and server's side complexity. Morfik allows these applications to be coded in a familiar language (one of several clones of popular languages like Pascal, Basic or C#), just like a common desktop application. This code will be trans-compiled into a server side application and a set of HTML and JavaScript files to be loaded in the client's browser. Hence, no web application or AJAX knowledge would be necessary.

From the demonstrations, I've spotted that the greatest feature of Morfik it's its familiarity with common Microsoft applications to develop the application. For example, a specific Morfik component is used to setup the domain tier, using an interface almost 100% identical to Microsoft Access'. Similarly, the web pages are designed using a WYSIWYG program, just like Visual Studio's interface builder. So any application used to the Microsoft's tools should be fairly suited to use Morfik without a two year-long course!

How does this Morfik advent leaves me? Well, I was planning to do something similar for my PhD, so now I must find out exactly what they plan to do, so my work won't clash with theirs. For instance, I'd like to know how do they split the computations between the client and the server - I'd like to give some adaptability to the server.

To add up, Morfik have a pioneers program, but they haven't released any usable version of the program for us to test it (not even demo-ware...). I'll just wait for further news on their site: http://www.morfik.com/

New directions, new homepage

2005-11-08T23:30:00.000Z

I've just reviewed my homepages, and it took me a while to get them
presentable... So here's my status, online-wise:

- I have http://360.yahoo.com/edy_in_carpathia - this is my personal website,
with access to photo albums, preferences and other non-interesting stuff :)
- My work homepage is http://edgargoncalves.no-ip.org - it has a set of projects I'm working on, and my professional information (like my resume)
- This site (last, but not least) will be my communication point, unless Yahoo releases a great API to post messages to it's blog. So, for now on, my PhD discoveries, researches and interesting notes will be posted here, for the world to see. Ocasionally one or other photo will invade this space, but I promise I won't have the motivation to do so (nor the time!)

So, on my next post, I promise to write about remote interfaces, my last research subject for the past weeks.

Ta-ta...

2005-06-13T23:56:00.000+01:00

The Vitruvian Man. Leonardo's most perfect piece, imho. This is also to tell that this blog isn't dead. In fact, it's sibling MSN Space has recently entered in a inevitable coma... But bear in mind that I have my own tech blog at www.esw.inesc-id.pt/~eemg/ - so this blog will be ONLY for personal/relevant photos and thoughts. why? because I have Hello. and because It grants me unlimited picture storage and a nice interface! :) So sit back, and enjoy the show...
Posted by Hello

Emacs, Gnus and BBDB

2005-04-25T13:34:00.000+01:00

Big day - I now have my BBDB decide where to send a mail from one of my contacts, either personal or mailling lists! My arch repository was updated accordingly (mail me if you want to know where it is).

BBDB exporting

2005-04-08T14:01:00.000+01:00

Alex Schroeder: Comments on 2005-02-22 Software: "Comments on 2005-02-22 Software"

mozdev.org - webmail

2005-04-07T19:06:00.000+01:00

Now THIS is handy! Finally, both Hotmail and Yahoo mails being delivered via a pop server, for free! The caveat is that the plugin only works via thunderbird - I wonder how much longer do we have to wait until a Gnus version is made? Download it now from mozdev.org - webmail: "mozdev.org"

ZipGenius: free zip unzip utility

2005-03-25T15:24:00.000Z

I found a good tool: ZipGenius: free zip unzip utility. File compression is finally easy and free, with no adware, no spyware, no ads, no money. ZipGenius is free and handles many archive formats: zip, rar, arc, ace, arj, cab, tar, 7-zip, zoo, tar.gz, gz, z, bz2, iso, nrg.: "If you need a compression utility, your search ends here: now you can count on ZipGenius and you it won't cost a dime to you, just because here you can find just FREEWARE programs."

2005-03-21T13:30:00.000Z

trying the current version of weblogger. if this blog entry has a title, I'll settle with it. :)

2005-03-21T13:16:00.000Z

Point one: with the weblogger package for emacs my subjects are nto appearing. gotta check why eventually. Point two: at INESC, I have emacs version GNU Emacs 21.3.50.1 (i386-mingw-nt5.2.3790) of 2005-01-30 on NONIQPC. Well, this has support for 'jpeg, and not 'gif. In my laptop, I seem to miss both of those image types! I wonder why??

2005-03-19T01:40:00.000Z

I made some updates to ERC nicklist module. Basically, it now shows me who's online and availlable, along with it's email. Also, the menu is functional! Now I need some icons to put before each nick, to tell me the away state and the connection type (msn, icq, irc,...). But it's something I'll so with more time. I'm happy to be part of the erc community, for now! I was testing weblogger.el, and I said I would say something about it. Well, I'm still using it for this post, so that means something! Basically, it uses the blogger API, so it can do a lot more that my email sending approach (however, I allowed for making a new post from an emacs region - I might merge that functionality, someday). Cheers!

2005-03-16T01:40:00.000Z

I've found yet another existing method of blogging here. This may be more complete, i'll just have to try it out more.

ERC and voice, with Bitlbee

2005-03-15T12:13:00.000Z

I have noticed that the voicing system is used in Bitlbee to let me know wich users are availlable, instead of being away, DND or simply NA. So, my next ERC modding will have to be a notification, to let me know what's the status of my so-called "friends".

Also, I have to check erc-bbdb to know the current MSN nick of a particular buddy. I know they do this on /WHOIS, so i'l just grok that and make it into a popup message in erc-nicklist, or something.

Btw, I just came across C-M-w in Emacs. The kill ring still amazes me! (Go on, check it...)

"Allows", and stuff like that

2005-03-13T14:55:00.000Z

I've just discovered that english papers don't like to use sentences like "it allows me to do", or "i was allowed to evolve...". Live and learn!

Blogspot posting from Emacs

2005-03-09T17:51:00.000Z

After setting up your blog to receive messages from a mail, eval this on emacs:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; Blogspot email sending:

(defvar blogspot-post-email "yourname.dummy_mail@blogger.com")

(defun blogspot (title url &optional text) "Opens a gnus msg buffer with the active region or the parameters passed in the body. Configure `blogspot-post-email' to set it up." (interactive "sPost title: sPost link: ") (let ((to blogspot-post-email) (subject (cond ((and title url (not (string= url ""))) (concat "" title "")) (title title) (t ""))) (body (or text ""))) (gnus-msg-mail to subject) (save-excursion (when body (raise-frame) (insert body))) (message "Edit your post and press \\C-c \\C-c to post it to %s." blogspot-post-email)))

(defun blogspot-region (beg end title url) "Call `blogspot-text' and inserts the active region as the post's body." (interactive "r sPost title: sPost link: ") (blogspot title url (buffer-substring-no-properties beg end)))

-- Edgar Gonçalves INESC-ID, Software Engineering Group (Technical University of Lisbon) Portugal

ICWI 2005: Call for Papers

2005-03-07T13:22:00.000Z

IADIS International Conference is an opportunity to write a paper on my final degree project. While it may not be a hot-shot conference, it's another published paper for my CV, along with the increased experience, yadda yadda... For details on the conference, see: ICWI 2005: Call for Papers

2005-03-05T15:31:00.000Z

These are my two girls. Taken at my sister's (the one in the right) 14th birthday!
Posted by Hello

RSS access to newsgroups via Google

2005-03-05T14:55:00.000Z

Ever wanted to read newsgroups in Firefox? Well, yeah, you can use google groups to achieve that. But what if you already read RSS/RDF/Atom news? You can read any group via the google groups feeder. See RSS access to newsgroups via Google for more information.

The Carpathia Blog is back!

2005-03-05T14:42:00.000Z

That's right. A new face, a new name - Carpathian Cloud - and a fresh motivation. I want this blog to have all of my daily discoveries (both social and techincal). But I mean to make some posts with some thoughts, some pictures I find interesting or relevant, and other stuff I may consider usefull to some viewer. I also have some new tools, and a ongoing idea to help me with my blog posting: Hello and Picasa are great to post images. JustBlogIt! is a Firefox extension that allows me to post something from any page, adding the link and title automatically. I also intend to make (or find, eheh) a mean to do the same from within Emacs. It would be great - and not that hard, I believe - more to come about this... Hope this can be of some use to someone. Cheers!

My computer is up and running again!

2004-03-01T22:09:00.000Z

After a quick tour with blaster, my win system became unstable. So, format c: /q did the job, and it is now fine (it was also full of shit, then...) So hey, new thunderbird nightly, and there goes another 7MB of international downloads :)

ImageMagick for MzScheme 103

2004-02-29T16:29:00.000Z

ImageMagick for MzScheme 103: With this baby we may make thumbnails on the fly, by making a nice surflet :)

Breeze Systems - Digital Photography Starts Here!

2004-02-28T20:07:00.000Z

I was advised Breeze Systems have a great app to deal with raw images from the Canon PowerShot S50 (my cam...) can't try it out right now, but I will. (ACDSee??)

Orion Application Server

2004-02-26T14:20:00.000Z

Orion Application Server This is a good resource for creating custom taglibs for struts/JSP's. Has more stuff, but I haven't checked it out...

NLO's Siemens C55 Page

2004-02-24T01:23:00.000Z

NLO's Siemens C55 Page Here are some tips about my current cell-phone (shoooo - it's supposed to be secret...) :)

2xExplorer home

2004-02-11T01:44:00.000Z

Go to 2xExplorer home and grab its newest child: xplorerÂ², itÂ´s SOOOOO cool. think norton commander + explorer + quickviewer + cuteftp and boom! xplorerÂ² :) (still in alpha by now...)

Hypertext marks in LaTeX: a manual for hyperref

2004-02-10T15:55:00.000Z

Click here to see the manual for this awsome package to latex - the default is those fugly red boxes arround links in the otherwise cute pdf's!! use something like \usepackage[pdftitle={my title},colorlinks=true,urlcolor=blue,linkcolor=blue]{hyperref}

emacs - a repository

2004-02-04T22:26:00.000Z

This emacs site has stuff about emacs-wiki, and a xml-parser for elisp (wanna read rss's? ;) )

Samurize.com

2004-01-28T03:16:00.000Z

Samurize.com - News weeeeeeeeeeeeeeee!

Pantheons

2004-01-27T18:16:00.000Z

This is a very complete site. Pantheons has a full list of gods, demi-gods and heroes of many different cultures, organized geographically. It's pretty good to learn something new!

WinT's Windows Modifications

2004-01-26T23:47:00.000Z

WinT's Windows Modifications: This is very cute! dialigs, toolbars and application modifications.

TinyWeb

2004-01-25T02:00:00.000Z

Ritlabs: TinyWeb. This is a win32 FREE, SMALL, http daemon wirh SSL tunneling... looks nice, but i'm not really needing it, now!

myRSS: PUBLICO.PT Channel Information

2004-01-22T02:13:00.000Z

myRSS: PUBLICO.PT Channel Information This is a good starting point to learn about rss...

A CSS editing mode for Emacs

2004-01-21T22:12:00.000Z

A CSS editing mode for Emacs check this out

AllegroServe - a Web Application Server

2004-01-21T21:50:00.000Z

AllegroServe - a Web Application Server hey, it's free now!! :)

The Common Gateway Interface Specification

2004-01-21T01:59:00.000Z

The Common Gateway Interface Specification> someday i-ll have a good look at this...

scsh.net

2004-01-19T21:29:00.000Z

scsh.net is the site for the Scheme Shell - yes, it's alive!! must give it a go...

CLiki (the common lisp wiki): IMHO

2004-01-19T21:13:00.000Z

IMHO, from onShore Development is a toolkit that provides facilities for building highly interactive Web applications. (using lisp, it seems!)

X-Men - Episode List

2004-01-18T15:14:00.000Z

X-Men - Episode List This is extensive, and very easy to follow, fully linked! start your kazaa engines!

Piled Higher and Deeper

2004-01-17T15:10:00.000Z

Piled Higher and Deeper Break time - great comic!

Running as a Service on MS Windows

2004-01-14T23:48:00.000Z

HAXIAL -- Running as a Service on MS Windows This is eveything you'll need - found out the firedaemon is free to use with ONE single service :(

FreeTranslation.com - FreeTranslation Tools for browser and web site translation

2004-01-14T02:39:00.000Z

FreeTranslation.com - FreeTranslation Tools for browser and web site translation really cool - add an applet to your website :)

Bitvise WinSSHD

2004-01-11T17:38:00.000Z

Bitvise WinSSHD sshd for windows. caveat: not free. (but "buyable" at astalavista...)

http://www.pmail.com/overviews/ovw_mercury.htm

2004-01-11T17:18:00.000Z

http://www.pmail.com/overviews/ovw_mercury.htm IMAP, SSL windows free server

FireDaemon :: Install and run Win32 applications as Windows NT/2K/XP/2K3 services

2004-01-11T14:05:00.000Z

FireDaemon :: Install and run Win32 applications as Windows NT/2K/XP/2K3 services emule, winamp server, and so on and so on...

DSL/Cable Webserver - Run your own webserver from home on DSL or Cable!

2004-01-11T14:03:00.000Z

DSL/Cable Webserver - Run your own webserver from home on DSL or Cable! hey - everything is here! :) someday i'll have my home pc do all that nasty tricks they mention...

And now, for something completelly diferent...

2004-01-09T23:32:00.000Z

MarklPirata - radio online (portuguese) eheheh

CyberCPU.net -- using sql in access db with asp

2003-12-31T02:11:00.000Z

might be handy someday :) CyberCPU.net --

i18n in asp.net pages

2003-12-29T12:22:00.000Z

Check InformIT.com : Programming >here to get a good howto and a good explanation of how globalization works!

New blog arrangement

2003-12-26T22:02:00.000Z

now, with archives :) not that hard, really. just a test...

2003-12-26T13:21:00.000Z

And finaly I made it: my first BB4win plugin that I am really going to use :) After the first revision I fixed two important bugs, and now it's pretty stable, for me. Link: BBbin 1.0.1

2003-12-21T03:07:00.000Z

Bart's PE Builder - Build your own Bart Preinstalled Environment (BartPE) bootable CD/DVD: wohoo! no disk? no problem! :)

2003-10-30T00:48:00.000Z

How the Grinch Stole Emacs A sweet and tender tale :)

2003-10-30T00:16:00.000Z

The preview-latex package woo-oooh! The only problem for windows is that "i need an emacs that supports images"... Maybe the CVS version, as they suggest, will do.

2003-10-29T23:16:00.000Z

Small HTTP server for windows - it's working, you're seeing it on this Blog! :)