Lisp macros, eval and packages

Today I spent about an hour trying to find out the source of a bug in the code for my PhD prototype. To make the context short, I need to have a class definition to be stored, and loaded afterwards. And I was using Gary King's time-saver metatilities:defclass* to reduce the required coding. The problem was when I wanted to load such a definition in a generic method, I was evaluating the definition expression - one of the acceptable uses of `eval', correct me if I'm wrong, is to interpret something that was just read (in this case, from a serialized string, from other servers). The code in question was something like this:

(defmacro load-data (name attributes)
  "Defines a domain concept, using metatilities:defclass* syntax."
  `(progn
     (metatilities:defclass* ,name (data-item)
       (,@attributes)
       (:metaclass profiled-metaclass)
       (:name-prefix  ,(format nil "~(~A~)" name)))
     (info "Data type loaded: ~A" ',name)))

(defmethod load-definition ((def data-item-definition))
  (eval (macroexpand `(load-data ,(definition-name def)
                                                     ,(data-item-definition-attributes def)))))

What I was getting as a result is that the auxiliary methods created as accessors by defclass* weren't being imported in my packaged, but in CL-USER. Hence, lots and lots of compiler warnings and subsequent errors when trying to use those accessors. I fiddled around, tried to see if I could remove the eval, but the solution, so obvious as I know can see, is quite simple. Just place your evaluated expression after setting the package. The fixed method is the following:

(defmethod load-definition ((def data-item-definition))
  (eval `(progn (in-package :dshow)
                ,(macroexpand `(load-data ,(definition-name def)
                                          ,(data-item-definition-attributes def))))))

Hope you can see this before spending too much time on a similar bug!

Quick Emacs nicity

I can’t remember from whose configuration I got this piece of code. But it makes my lispy screen a little bit cuter, by turning all “(lambda” into the greek character. I tested on aquamacs, and on the Windows GNU Emacs, and it works. If anyone knows the original author of the idea, send him/her my thanks :)

;; Make lambdas appear as the character » (saves space!!)
(defun pretty-lambdas ()
  (interactive)
  (font-lock-add-keywords
   nil `(("(\\(lambda\\>\\)"
          (0 (progn (compose-region (match-beginning 1) (match-end 1)
                                    ,(make-char 'greek-iso8859-7 107))
                    nil))))))
(add-hook 'lisp-mode-hook 'pretty-lambdas)
(add-hook 'emacs-lisp-mode-hook 'pretty-lambdas)

On iPhone SDK and its pretty icons on tab bar items

After having made some experimentation around the iPhone SDK, I have now a simple - but useful - native application. I focused on the functionality, and up until now, I had a tab-bar with no icons. But that's not the Apple-way to present things. iPhone apps are pretty, and have simple but elegant icons with a gray tone that turn blue when selected. So i looked upon a library/bundle for some of those standard buttons, and found... none! Ok, no worries, let's find out how to make them. Wait. There isn't a specification for them! Yes, it's a shame, but Apple's iPhone SDK documentation is still a bit unfinished. The good news is that what is done, is easy to read and helpful. I ended up looking at other apps resources, found out they were 32x32 png images, and figured they had to have at least an alpha channel outside the image itself. So I fired Gimp, made a plain brush doodle and removed the background white, saved and tested. And it simply works! All colors are turned into the desired gray tone, and automatically swap to a smooth blue with a top glare when selected. Conclusion: the trick is to experiment. Chances are that what you want to do is *that* simple!

Milky love for lispers

I've just updated my "Remember The Milk"-talking lisp library. Now you can access their API with a simple CLOS-based object model.

You can fetch the latest version by direct download, SVN and/or by asdf-install from the project page, where I have the important updates:

Recent Changes

* Added an object bridge to make the API even more integrated with the language.

* Methods and functions were coded to be easier to manage a local state of the RTM user data.

Future Work

* Improve synchronization and bandwidth usage by fetching only recently updated data;

* Provide offline work for the local data

Having a cow aiding getting your things done

I have finally stopped to upgrade my Remember the Milk (RTM) usage to something more GTD-ish (for David Allen’s “Getting Things Done” approach). I already knew about RTM’s organizing abilities. Succinctly, we can have tasks in our inbox, in lists (hard to move from, usability-wise), and have them assigned one or more tags, easily exchangeable and searchable. What I was looking for was a way to have my tasks split into contexts (like calls to make, errands to do, things to do at the computer, things to see, read or listen). Also, I need to see what I can do next, what I have to do someday, and things I must hold on pending, waiting for something/someone. With these two axis for plotting my tasks, I can add stuff to RTM as they come up in my life (via many different ways like their website, Twitter, quicksilver, a Dashboard widget, iPhone...), and they will get placed into Inbox. At the start of the morning, I have to process that list, placing tasks into a proper context List (I prefix those with a @, to visually identify them at a glance). I also tag each task with a “next”, “someday” and “waiting” tag. The final visual output is clean enough for me to feel organized:

Let’s get the webpage hacks discussed first. It’s simple enough, but important for me. The default tag strip wouldn’t do it for me. So I got this amazing Greasemonkey script, called A Bit Better RTM. Three simple features that make all the difference!

Step one, projects. I followed many advices at the RTM forum, and created a list called Projects, where I place project titles (prefixed by a dot, for visual identification), and tag them with a “p-PROJECTNAME” tag. This way, I can easily click that tag and access all tasks (from one or more contexts). It’s really handy.

Step two, smart lists. These are simply saved searches, and we have a decent set of search operators to use. Here’s some I have installed:
• ->Next: (tag:next AND NOT dueAfter:"1 week of today") OR dueBefore:"1 week of today"
• ->Someday: tag:someday
• ->Waiting: tag:waiting
• ->Tickler: NOT due:never
• PhD: tag:PhD OR tagContains:phd
• Fun: tag:entertainment OR tag:game OR tag:movie OR tag:night
• unTagged: isTagged:false OR (NOT (tagContains:"next" OR tagContains:"someday" OR tagContains:"waiting"))

Tickler is a must see every day, to keep myself from forgetting due tasks. Phd looks for tags like p-phd-thesis, thus looking for all projects related to my Ph.D.. unTagged is a reminder tool. It often happens that I move a task from inbox into a given list, and forget to tag it (all tasks should have a next/someday/waiting tag on it).

Don’t forget that tasks can easily be assigned a priority with a 1-4 key press. Also, I have locations set up so that they appear in the task cloud. This way I can look at all the calls I have to make at the office, for instance.

So with this I feel almost at ease. But I’m kind of a control freak, and I like to add tasks in the easier way I can. After having played with Firefox’s ubiquity to write Twixfer, I started jotting down a quick way to add a task to RTM. The initial ubiquity command was usable, right until it popped up a new tab with a RTM task properties form. I didn’t need that. But someone felt the same, and spared my the effort. rtm-v2 is a command set that logs in, and gives proper feedback of your RTM lists and tasks. If you’re into speed task jotting, it’s worth a try.

That’s my rtm tips for today. Do it like me? Have a better system? Drop me a comment!

Love Buxfer. Use Twitter. So why not Twixfer?

Today I sat down to find out a different use for Javascript. Browsed through my everyday task list, found out a couple of candidates, and targeted Buxfer transaction creation. I'm also growing fond of Ubiquity, for the Firefox browser (and is now the most important reason I'm not moving to Google's Chrome, but I digress). This post, despite being about a Buxfer contribution, will be more about the Ubiquity development process I underwent.

So let me explain what I aimed for. Ubiquity allows to enter text controls, with auto completion in a HUD over the Firefox window. So I wanted to do something similar to what Bruno Lopes did with his BuxferSubmit Dashboard widget (for Mac OSX) - plain simple addition of a Buxfer transaction, with a description and an amount. All I have to do is (after having Ubiquity installed and running) to setup a Javascript file with a piece of code I'll explain hereafter.

The heart of a Ubiquity command is defined with the commandfunction CmdUtils.CreateCommand(argument-object). This is, in fact, the only call you have to make in that file to set things up. So what's in an argument literal object? the basic attributes, like the name of the command, it's description or the icon to identify it (quite useful!). But there are a few arguments that deserve special attention

• execute. This must be a function with no arguments that is responsible for doing whatever the command is supposed to do after <return> is pressed.
• preview. This is the function that is called at each key press on the Ubiquity HUD, to compute the string that appears to help the user find out what the command is going to do.
• takes. A description of the words the command receives, and a type for them. Each type can be specified by hand, but several are already provided, like noun_arb_text for arbitrary text or noun_type_date for a date string.
• modifiers. These are words that offer secondary semantics to the command, and are specified just like the “takes” argument elements. Think of examples like “list-birthdays” for today's birthdays and “list-birthdays for next week”. The modifier “for” would be assigned to “next week”, and a special type can be setup to parse the string and compute the date from today.

To find examples about how to use the CreateCommand, load the following URI in Firefox: chrome://ubiquity/content/builtincmds.js (obviously only after having ubiquity up and running).

That's about it. What I found out was that the presence of a preview functionality can often be more useful than the execution of the command! Think about a currency conversion, or an algebraic expression calculation. I don't want to perform anything upon a <return>, except perhaps have the result copied to the clipboard. What I really want is to have the result appear in the HUD and change dynamically as I keep filling in the command text.

Let's move on to the implementation itself. One thing I found out quickly was that the Buxfer's APi needed a username and password to be able to post a transaction. That would be the case in most web applications I know about. But this API doesn't offer, for now, any means to open a page to ask for credentials - nor would I desire that, since I'm typing a textual command to keep the input at a minimum! Ubiquity doesn't have a thorough documentation, yet, and browsing the javascript XUL files within the chrome “filesystem” of a Firefox add-on is not something I like to call user-friendly. Why is this important? I thought the command could take a peek into the Firefox's secured passwords and use the Buxfer, without the user having to care a bit about it. But then again, an exception would have to be made in order for first timers be able to use the command… I was about to set up a proxy web service in Google's App Engine when I remembered two happy coincidences:

• Buxfer allows Twitter messages to be sent with the same syntax as those we can send from an API (or from a cell phone).
• Twitter has a nice API, that does exactly what I was aiming for: it points it's requests to they're site, and if you're not logged in, it asks for your credentials using HTTP Basic Authentication. This way, Firefox aks you if it may remember the password, and actually does it :)

The result: I ended up with two commands, one using the Buxfer API, and another using Twitter's. But since Twitter's the most user friendly (easier and faster to use), I'm switching my transaction input to the brand new Twixfer (It's weekend, my imagination needs some more rest to come up with a better name!). Feel free to take a peak into the homepage. The source consists of a couple of files, so I thought a Google Code project would be over-killing it… But if this thing starts to get any sort of user input/contributions, I'll gladly move it there. For now, leave comments at the Twixfer site. As usual, feel free to leave feedback about anything.

[EDIT: Today I had a tip from a gentle soul about retrieving password from Firefox's session. So Buxfer API is usable by itself, now. Check the same link and update your command, if needed. It has lots of cleaning up and code re-factoring done. Nevertheless, it was fun to toy around with the Twixfer concept!]

Getting Things Done - the flow-chart

Here's David Allen's graph for Getting Things Done (GTD). Though to use, but has nice results!

Moooing away in Common Lisp

I'v recently found myself needing to use a task manager to make some bulk operations (to be more precise, I wanted to copy all of my tasks to a device with no built-in synchronization mechanism for those). I happen to make use of the great Remember The Milk Web application (I've previously worked with Emacs Planner, but I had to drop it in favor of a more mobile version of my life).

Enter Common Lisp. Perhaps the fastest way to do some scripting in my mind, but I've yet to make a library to use a Web-based API. Turned out it was quite clean, and in about 5 hours of work I had the whole API grokked, coded, tested and hosted in GoogleCode here. (I looked into cl.net, but found the google's approach to hosting much more featured for the fast start I was looking for. If this has enough momentum I can look into it again, with more time - the main advantage would be asdf-installability, I suppose.)

I already had experience making project "skeletons" (asdf project file, package declaration, etc.). So the hardest thing I had to do was look into the web API call mechanism and into the md5 encryption. If you don't need the detailed description, tha answers are, respectively, Drakma and Ironclad. ( I also use cl-json to parse RTM's responses into a lisp-friendly format.)

To call a Web-based method, we need to make an HTTP request, and be able to read the response. using drakma (asdf installable and loaded in the usual way), all I had to do was something like:

(http-request rtm-api-endpoint :method :post :parameters '(("param-name" . "param-value")))

With some parameter abstractions, RTM's was easily tamed. A call to one of their methods is now made with the following macro:

(defun rtm-api-call-method (method &optional key-value-pairs &key with-timeline with-authentication (format "json"))   "Calls `METHOD'.  - Optionally passes pairs of strings in `KEY-VALUE-PAIRS' in the form of (`PARAMETER' . `VALUE').  - Keyword `WITH-TIMELINE', if not null, allows the method to be called within the `*CURRENT-TIMELINE*'.  - Keyword `WITH-AUTHENTICATION', if not null, allows the method call to be authenticated with a valid `*RTM-API-TOKEN*'.  - Keyword `FORMAT' is one of \"json\" (the default value) or \"rest\", and specifies the server reply format."   (declare (special *current-timeline*))   (let* ((parameters `(("api_key"    . ,rtm-api-key)                        ("method"     . ,method)                        ("format"     . ,format)                        ,@(when with-timeline                                (with-timeline                                  `(("timeline" . ,*current-timeline*))))                        ,@(when with-authentication                             `(("auth_token" . ,*rtm-api-token*)))                        ,@key-value-pairs))          (api-sig (compute-rtm-api-sig parameters)))     (multiple-value-bind (result)         (http-request rtm-api-endpoint                       :method :post                       :parameters `(                                     ,@parameters                                     ("api_sig" . ,api-sig)))       (let* ((response (json-bind (rsp) result rsp))              (stat (assoc :stat response)))         (cond           ((string=  (cdr stat) "ok")            (rest response))           ((string=  (cdr stat) "fail")            (let ((err-info (cdr (assoc :err response))))              (error "RTM error code ~a: ~a~%" (cdr (assoc :code err-info))                     (cdr (assoc :msg err-info))))))))))

With this (rather big, for the sake of readability, or mayhap due to my lack of experience in mental pretty printing) function, I define a call to an RTM module simply by coding:

(defun rtm-api-tasks-complete (list-id taskseries-id task-id)   (rtm-api-call-method "rtm.tasks.complete"                        `(("list_id"       . ,list-id)                          ("taskseries_id" . ,taskseries-id)                          ("task_id"       . ,task-id))                        :with-authentication t                        :with-timeline t))

The only call in the first function definition that cannot be easily guessed how they work is the compute-rpm-api-sig-parameters. This is the function that performs the algorithm required by RTM to encode all parameters and guarantee authenticity in each request (along with a masked API key). Basically it does some mambo-jumbo concatenation with each parameter's name and value, and then performs an MD5 to get the equivalent hash code. Easy, right? But I'm a lazy programmer. That, and I truly believe there's no point in reinventing the wheel, so I looked for encryption functions in Lisp, and I found Ironclad, a collection of most of these functions in Lisp. So producing an md5 functions was as easy as the following:

(defun md5 (string)   "MD5 uses ironclad to encode `STRING' into an hexadecimal digest string."   (ironclad:byte-array-to-hex-string    (ironclad:digest-sequence :md5 (ironclad:ascii-string-to-byte-array string))))

And that's about it. The rest was toying around, the API works perfectly (kudos to the RTM dev team!), and I was thrilled to have my tasks exported in a jiffy. If you want to find more info, feel free to snoop through the code (it's quite easy to grasp, I think, and relatively small), in the Google Code site: http://code.google.com/p/rtm-lisp-api/ . So now, what would you, gentle reader, do with this API? Drop me some comments with your ideas.

Until the next time!

Cocoa development, using a Lisp bridge

Let's get a bit away from my PhD for a while. In my spare time, I've been looking at some graphic libraries to make some applications. In the past I've looked for Operating System portability for those apps. I've also delved into the Web world for interfaces. Right now, I've settled for making pretty looking gui's. If you add my new acquisition - a Macbook Pro, my first mac ever! - and I obviously had to try to make a MacOS Cocoa application. They're fashion, they're usable, they mingle with the rest of the desktop manager, and I was just tempted to try to use Lisp to make something like it.

Fortunately, this wasn't so hard, has OpenMCL has a great step by step tutorial (to be taken with a few glasses of water and with the Cocoa / Interface Builder tutorials made available by Apple). This tutorial has detailed instructions of how one can build the Interface with the WYSIWYG tool that comes with Xcode. Then it tells you how to setup the connection points between the interface and the application code. These are called outlets, for the variables in the code that represent a given object (so that you may, for instance, retrieve the text in a text field) and actions, for the functions that are called when something happens in the interface (e.g., user click in a button). After creating the necessary objects (basically you must make them belong to a specific metaclass in order for them to be exported as an ObjectiveC class), all you have to do is call a function that binds together the compiled code, the Interface files (in the nib format) and places a nice application bundle in the directory of your choice. It can't get any easier, you think. And yes, you may code interface changes in you lisp code. But it's extremely easier to follow in Interface Builder guidelines and helpers... You just can't go wrong!

You may find the tutorial at the OpenMCL wiki: http://trac.clozure.com/openmcl/wiki/AppleCurrencyConverter

User interface paradigms for data search and offline data access

I've stumbled upon this interesting and illustrated article on data lookup user interface design patterns (on both web and desktop applications, finally we seem to be starting to forget there's a difference!):

Here are two design paradigms for handling large amounts of data, not to be confused (or combined) as web design meets desktop in rich Internet applications.

[From Seek or Show: Two Design Paradigms for Lots of Data « Theresaneil’s Weblog]

The post describes two main groups of patterns, based on the way a user chooses what he/she wants to search (pictures linked from the article mentioned above):

• The Seek Paradigm - The user has to input a search criteria in order to access some search results.
  
• The Show Paradigm - The user is presented with several search results (or categories), and he may drill-down the search by clicking on the desired categories or selecting certain results to eliminate unwanted items.
  

This got me thinking in terms of offline work, and I can easily qualify both approaches as orthogonal to the online vs. offline decision. It is merely a user interface option, which is a good thing. To understand this, picture a lookup for book titles on a library. To go offline, one would have the client application to fetch them all from the server, thus making both kind of interface patterns possible without contacting the server. Similarly, if we wanted to know how many clients had already rented a book (assuming the algorithm implied a run through each client's details to look for their rental - I know, it's not the smartest way to do that, but it's only an illustrative example!), the client wouldn't be able to fetch all the other clients' details, hence not being able to use any of the search paradigms.

So the effort that has to be done, from the application logic design point of view, is related to how the services (those accessed by the user interface) access the information stored in the database. Let's draw some pseudo-code* for the book rental example above (the second example).

• Approach 1: The all-powerful service. This is the kind of service we would have written up until a few months ago, considering any sort of offline execution to be an academic dream or an edgy unfeasible mostly unsuccessful experiment. Basically it assumes that it is being executed with super-user privileges, hence able to access all stored data at any time with no constraints whatsoever.
• Aproach 2: The helpful, aware and conscious service. This is the service that knows that its code may be executed in many places, and so it tries to make a scarce usage of the stored data as much as it can.
• 
  

Using approach 1 we could code a service like this:

(define-service get-number-of-book-rentals (book)
  (let ((all-clients (get-storage-data "clients"))
        (result 0))
    (dolist (client all-clients)
      (set result (+ result
                     (client-book-rentals client book))))
    result))

As you can see, while this would work perfectly on the server, the client would never be able to to the (get-storage-data "clients")call, given its lack of permission to fetch them all to the client-side storage. This is the sort of services that must be thought over and rewritten - or, better yet, automatically translated to something more useful!

By using approach 2, we would instead code something in the lines of:

(define-service get-number-of-book-rentals
  (:inputs (book)
   :storage-data (books-rent-by-all-clients (get-total-rentals-from-clients)))
  (let ((number-of-books 0))
    (dolist (client-rentals (filter book-rent-by-all-clients 
                                    :field "book" 
                                    :value book))
      (set number-of-books (+ number-of-books client-rentals)))))
    number-of-books))

Here it's clear that the service knows what data is going to be needed from the storage. With this information, the server can fetch it for the client when it wants to disconnect, and the client is now able to perform the same functionality without having to run through the clients (read, without having to have access to all clients' personal details). Relevant, noteworthy changes between the examples:

• the service has to be declared with more information (like the storage-data requirements)
• the storage data requirements code is always performed on the server. The client counterpart consists of a lookup on the client-side storage for the previously fetched information. This way there is no risk of information not being available.
• If the service is to be run online, it also runs equally well, with no performance penalties.
• There must be either a mentality change or a tool to translate services made with the approach 1 to the approach 2!

I'm already laying out some code to achieve something like this. Expect news in the following week or two.

*Disclaimer - despite being in a lisp pseudo-dialect, all examples could exist in the language of your choice. It's really just a matter of personal preference!

The search for functionality dependency

Having had my PhD thesis proposal (previously presented here) approved by my university, it's time to move along to the next step: thorough description of the solution I propose, followed by its implementation.

The first important aspect I need to address is the grouping of functionalities that may be executed by a given user while offline. The ideal situation would be to pick up a specification of a Web application, then automatically produce a graph of available functionalities, with their dependency hierarchy. From here, the developer (or even the project engineer) could inspect what would be desirable to be performed offline, according to the requirements artifacts. Since we're also aiming for ease of use, on the produced tree, it would be nice to see the characteristics of each functionality, namely when related to access control permissions and data dependencies. This way we could gray out some of the available functionalities that could never be performed offline (e.g., we may want to force every functionality requiring a specific role to be accessed online only). By imposing restrictions, one could easily make a map of he needed functionalities. It seems almost to easy. What we're forgetting is how we're going to get the dependency graph. Yes, I intended to get those via a workflow graph. But let's face it, most Web applications are built without a prior workflow specification artifact ever seeing the light of day. Specifically, my case study doesn't have one (yet, however). So how shall I do this?

• Write the aforementioned workflow specification by hand. This seems a lot of work to ask of the project development team, specially since all they want is some functionalities (they don't know exactly what or how many in a low-level, only the high-level use-case names are written in the requests, usually). It's also not possible to write it automatically, since it needs domain-specific information, usually more than what is already present (albeit very scattered) in the existing application. So this is a no-go.
• Code walking. If we inspect each interface user action and the accessed data and services, we could obviously draw a graph. This however poses other problems. Are all programmers doing the right thing? It's not that uncommon to write services that check on other services without needing them, just to see some futile/temporary information, and forgetting about it later on. I know that doesn't happen in a perfect world, but "real" and "perfect" don't always pair along! But there is another show-stopping issue with this approach: we would have to start the "walk" somewhere, so we need to know all the starting functionality calls, along with the list of services itself.
• User experience tracing. This is quite interesting in theory, if we ask users of several roles to perform their tasks, we can obviously trace the execution flow perfectly by reading the logs only. But we have no means to guarantee the users are going to access all the functionalities they are entitled to perform in our tracing period. That is only possible in a closed, controlled experience, by having users following scripts. But this kills the main purpose - those scripts are as hard to write as the workflow description itself, so it's of no use to us.
• Functionality reification. By reengineering the application to have the functionality calls represented as objects, we can have the immediate dependencies graph automatically by scanning all user interfaces and listing the graphs of all calls triggered by buttons, links or other events (XHR based, for instance).

I'm going to put my efforts on this last one. There are other advantages with this approach, being studied by some colleagues of mine at INESC-ID and in the Fenix development team. The main ones are related to he presentation mechanism allowed - the organization of the Web application interface can be composed with such objects. I'll get into more details afterwards.

As usual, I appreciate any comments, tips or any other kind of experience sharing from you, dear reader!

Microsoft's Occasionally Connected Smart Clients

There are two broad strategies for designing smart client communications: service-oriented and data-centric. When you have determined which of these to use, you need to make some fundamental design decisions to allow your smart clients to work offline. In most cases, the clients should be designed to use asynchronous communication and simple network interactions. Clients will need to cache data for use when offline, and you will need a method to handle data and business rule conflicts when the clients go back online.
[From Chapter 4 - Occasionally Connected Smart Clients]

This is a good overview for what I'm going to handle. Microsoft wants its PDA's, and other Smart Clients to be able to perform tasks while disconnected from the server. I want regular Web Applications to the the same, so I'm making the client browser just lie Microsoft's Smart Client. Let's overview their main points:

1. Approach: MS distinguishes a Data-centric and a Server-oriented approach to disconnected systems. The following graphic (taken from their article) illustrates that while data-centric approaches are not scalable due to their local replication of the entire database, the service-oriented approach also suffers from the lack of business logic (thus relying on the local cache without letting the server know it is not being contacted. We can envision server updates to this interface and/or business logic with painful consequences for the client... My approach is a mix between those two, for I want to design a client that is able to fetch a server module with both UI and business logic, and the necessary data for it to work. However, because we want the application to update the workflow, all client actions will still be postponed to the time that a server is reachable.
   
2. Conflict types: I've previously detected that all conflicts aren't the same. In fact, whenever someone edits a previously existing data (e.g., a student changing his email address), there is a simple data conflict, as the information on the client differs from the server's version. But this is a different case from an edition to a already deleted data (e.g., the same student tries to answer a question in a quiz that has already been removed by the teacher). MS article refers to the first case as Data Conflicts, and to the second as Business Rules Conflicts. Note that data conflicts may not be trivial to reconcile - consider a situation where two different entities change the same information, one offline, another online. Both want to change it, only one will succeed at first, but the offline client may need to merge its data, after having it sorted out with the other user.
3. Locking mechanism: while a pessimistic approach would minimize all sorts of conflicts, it is obviously not scalable to a widely shared data-set. So with an optimistic approach comes the need to synchronize and reconcile conflicts.
4. Reconciliation types: MS assumes the same three locations to do the reconciliation I've detected so far. If the conflict can be automatically merged, it will take place on the server. If not, it may take place on the client (with a simple user interaction dialog), or it may require the launch of another task that will contact other parties (e.g., two students posting the right answer to a question, needing the teacher input to opt for one of them).

The article further discusses CRUD operations as the ideal services to have in these situations (i.e., complex services would make client-side data synchronization harder and harder), and propose some strategies to handle task-based functionality dependencies.

While this is the closer I've read so far to what I want to do for my PhD, it's still lagged behind, as I propose a methodology to adapt existing applications (e.g., pick up existing web services and build from there), and I propose a conflict - and resolution mechanism - description that seamlessly integrates with the remainder application.

QUATIC07 - the humanistic version of the story

On the last post I mentioned my findings from QUATIC07, and specifically from the input received from the software engineering doctoral consortium workshop. Now it's time to clear out what I learned from the whole experience.

Let's start this thing to mention this is my second event of this sort. I've attended to CAPSI 05, to present a paper entitled "Desduplicação sobre um conjunto de nomes próprios" (in english, "Duplicate removal in a set of first names"). The paper was presented by Helena Galhardas, that despite being the third author - my teacher of decision support systems at the time - she had both the recognition and the experience to make a proper presentation without the beginner's jitters...). So, despite having a published paper, I hadn't present any, so far. Enter the QUATIC event. I'm now faced with the opportunity to show my PhD expectations to other students and teachers of the same area. And present a poster to the entire academia/industrial audience on the conference's hallways. And have my thesis proposal published as a paper on a IEEE proceedings! I must say I was pretty excited about the whole deal!

Day one. I arrived at the conference to present the thesis to the doctoral workshop, SEDES. The conference was in portuguese, and the first interesting fact I noticed was the geographical tendencies of the participants: I was the only one who lived/studied/worked below the northern part of portugal. The rest were all from Coimbra, Aveiro, Oporto, so Lisbon (!) was a strange city among them :) But everyone was really looking for a chance to get feedback on their work, so there wasn't much time nor motivation for picking around that! Well, after receiving my reviews, I met personally one of the reviewers, and found out he is going to be part of my PhD jury. I had lunch with all of them, and proceeded to the remainder of the thesis discussions. And that's about it.

Day 2 and 3. The poster. I placed the poster on a wall in the hallway, and got a nice desk to sit at. This way, I got to show some supporting images to some enquiring passerby. I made an english 10 minute presentation about it, to the industrial world, and got some positive feedback on the subject (the technical questions were a bit far from what I was expecting; the academia is surely more prepared to criticize my work, as it is still a novelty to the others). On the last day I got to see a teaching perspective on software engineering. Miller's presentation about TSP and PSP were most enlightening and entertaining. The day ended with a dinner in a Brazilian restaurant with live music. I got to mingle a bit with people I had met but rarely had the chance to talk to. It was fun, but I have to admit I was pretty tired. I went to bed and slept for a good 7 hours, before waking up to finish another presentation (7 hours seemed a lot, as I had been sleeping from 3 to 5 each night until then!)

Last thoughts. I had a great time, the preparation was quite helpful in organizing my mind schemes, and the feedback was the cherry on the top of the cake. But now I need to evolve my work so that I can go to a IEEE well known conference - not that this wasn't a good one, but I need some heavyweight recognition when I'm facing a fulminating jury! Advice for the PhD newcomers? Don't miss out opportunities like this! It can be expensive, obviously, but there are organizations able to fund your research and your publications and presentations!

Going back to Emacs (is there any other TeX editor worth reverting to, when you're an emacsen?)

QUATIC07 - the academical side

I've attended the last edition of the QUATIC conference, a three-day event on Universidade Nova de Lisboa, in Portugal. My main motivation was to participate on SEDES, a satellite workshop on Software Engineering PhD students. The purpose was to discuss ongoing PhD thesis and receive early feedback on them - while checking out other possibly related ideas and suggestions from the other participants.

I presented my PhD proposal, as it was by the end of July of the present year. It was a good experience, I received lots of appraisal for my chosen thematic, and also lots of critiques on the details of the thesis, my way of explaining things and my presentation. So lets start from the beginning, go straight to the end, then stop! :) (bored readers be warned, this is going to be a long post!)

My PhD thesis proposal is entitled "Offline execution in workflow-enabled Web applications". My goal started a bit broader, at a time where AJAX was starting to make its entrance in the development show-biz, I wanted to make desktop-like Web applications. Soon enough I realized I was foolish, because tens (hundreds?) of tools were immediately born to make Web Apps. look pretty, with lots of drag and drops, image overlaps, you name it. When something like Yahoo! releases their widgets, I can confidently stop trying to do something like that - I'm one student, they are one fully employed - read "payed!" - set of experienced developers. So I refined my goal and picked a real request - to make a Web Application run offline, without requiring external applications and without loosing any work. To do this I identified three main areas, or sub-problems:

1. Workflow decomposition into offline performable modules;



2. Offline client generation, and its functionality



3. Offline work reconciliation with the server

And this was what I had in mind at the time of the short paper submission. I was also invited to present a poster on how my work would affect the industry, with a short talk within the main QUATIC conference. That, off-course, set the stage for more interaction, as my poster was on a mandatory corridor to all participants ;)

Later on, I collected a set of information (from both research and opinions on the conference) that made my thesis improve itself. I'll try to present them here, as succinctly as I can:

• Workflow decomposition is a relatively known problem, and people like van der Aalst have been working on Colored Petri-Nets (CPN) to work it out. The problem is now reduced to the question "How do I tell if I can perform this activity if I go offline now?". This is not as simple as it may seem, as I must ensure that there is a possible execution flow that won't require other entities online interaction. I also must ensure that I can bring all required data offline, to my laptop - and issues like access control or the amount of information arise here.



• Offline persistency on a browser is now the purpose of tools such as Google Gears. Client-side generation using a high level language is also attacked by tools and frameworks like Morfik, XML11 or Google Web Toolkit. My problem is now to unify the specification of both online and offline tools into one. Later on I might also have to worry about technical details like client application transfer, shrinking, and a workflow-engine client clone, but not for a few months, for sure!



• I was told the synchronization scheme I was looking for was already done by the nice guys from Redmond. The Occasionally Connected Systems framework is supposed to handle a local database (SQL Server Everywhere) and all the asynchronous messaging it takes to handle eventual internet failure, small period disconnects, etc, and that would include a nice and efficient merge of all offline data. While this may be true, there are a couple of issues that allow me to stay on the innovative side of the research: 1) The framework is intended to run on desktop-based applications. I assume that eventually Silverlight-based applications could be supported (as they grow the run-time virtual machine and bloat them with even more .NET libraries). But I want Web applications to do the same, and that's not been abridged, as far as I can tell. 2) The OCS framework is able to merge database information. But the types of conflicts I'm targeting are the high-level ones. I'm not interested in the last-edition time-stamp that is different. I'm looking for the product that cannot be bought because the store now is set to open from Monday to Thursday and I can only get there on Fridays. To make things clear, I'm not interested in a data merge algorithm, but in the elegant and simple way to integrate the specification of such kinds of conflicts - and their resolution strategies - within the application domain model and workflow definition.

This was a nice set of input. I was pleased. But I had some other things to fix, not so technical but about my approach to the whole PhD:

• My sales-pitch - I need to fine-tune my goal statement into something that says "I want to describe a specific solution to a specific problem", and not "I want to solve al world's problems in two years, yay!!".



• I also must get a supportive example. My example has fallen in the field of the non-problem - that was what I got by making it so simple I could explain it in a corridor walk... So the suggestion is to find a concrete requirement in a concrete (aka, real) system, describe the requirement, describe the problems in the current approach and conclude by suggesting how do we would want to do it.



• I had to change a lot in what I was expecting to produce. Since I'm aiming for a development methodology (and the necessary framework or tools), I had to clearly state the inputs, outputs, all stages, the applicability/restrictions, the target audience, ad so on and so on. It's a shame there is no standards related to methodology description. I could certainly take advantage of one, now!

I'll post later about my personal experience on the conference - something from my human side :), and also to publish a copy of my thesis proposal. Here's how the poster looked like:

By the way, I have to say I consider myself fortunate, as this short paper is now published in a IEEE Proceedings! So, do you have any more advices, or experiences you'd like to share?

RIA and Flash

Last Wednesday I went to a seminar on the application of Flash technologies to produce Rich Internet Applications. Adobe has certainly been busy for the past year, as we are far away from the "banners and intros" main purpose for Flash. Two main names pop-up for discussion: Flex 2 and Apollo. Flex addresses the issue of producing web applications with Flash and ECMA Script (the ActionScript variant), in the same manner one would develop a HTML+Javascript solution. The coding language is named MXML, and allows to mingle the UI and the client-side scripting. The main advantages are the Flash remote communication capabilities (either via web services, flash remoting, and other RPC alternatives) and the Flash multimedia embedding. It's quite easy to add a video playing in a given canvas! As far as coding artifacts are concerned, all we need is at least an .MXML file, like the following:

<?xml version="1.0" encoding="utf-8"?>

<mx:ApolloApplication xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical" cornerRadius="12" alpha="0.7" borderStyle="none">
    <mx:Script>
        <![CDATA[
            import mx.collections.ArrayCollection;
           
            [Bindable] public var people:ArrayCollection = new ArrayCollection(["one", "two"]);
           
            private function doAddClick(_name:String):void{
                people.addItem(_name);
            }
        ]]>
    </mx:Script>

    <mx:TextInput name="firstname" text="Type your name here"/>
    <mx:Button label="Add it!" click="doAddClick({firstname})"/>
    <mx:ComboBox dataProvider="{people}"></mx:ComboBox>
    <mx:FileSystemTree/>
   
</mx:ApolloApplication>

Yes, this is purely XML. One can add up pure actionscript files, wich, by the way, has some advantages over javascript (it's an OO language, for instance). The main drawback (yes, there is one) is the IDE is not free (unlike the compiler and SDK - those are FOSS! Also, there's no DTD, or XSD available to use in XML development editors, so we will probably have to stick to an Eclipse plugin (paid Flex Builder) until an open source alternative comes around. RIA has been forever tied to the sandbox concept: like all web applications, one can only access files that have been uploaded by the user in the present session. Another restraint is that a browser is needed to run the application. Here comes Adobe's Apollo. Take a Flex application, pass a short application XML descriptor, and with a couple of command line instructions you have a compiled and packed installer to a standalone windows/mac executable with the MXML contents. But they didn't stop there. Adobe also wants to catch up current HTML/JavaScript/AJAX developers, so Apollo also makes identical standalone applications out of pure HTML+JavaScript sites. But that's not all. to our coding pleasure, Apollo lets us access all Apollo's ActionScript facilities from Javascript, and vice-versa! This means one can code a javascript handler to access debug tracing, common dialogs, client-side file system, loke in the following example (taken from Mike Chamber's blog):

//called when button is pressed to select a file
function onFileClick()
{
 //this will trace out the string to the command line
 apollo.trace("hello");
 
 //get a reference to the desktop
 var f = apollo.flash.filesystem.File.desktopDirectory;
 
 //listen for the select event
 f.addEventListener(apollo.flash.events.Event.SELECT, onFileSelect);
 
 //open the browse dialog
 f.browse();

}

Again, the SDK is (yet in alpha state) FOSS, but the IDE consists in an extension to Flex Builder, and thus is paid. But the technology itself seems quite promising. See Adobe Digital Editions for a quite nice real-world example. In short, I'm impressed with Adobe Labs, and it seems there's more to come in the next couple of months!

Free Lisp Executables in Windows

One of my latest projects has led me to think about making windows executables for a typical database application. The main requirement was using free software to make this project, so here's my starting point:

• Clisp, 2.41 (2006-10-13) (full, with readline support)
  
  
• ASDF
• IDE: Emacs and SLIME (latest, from CVS at the date of this post)

The first issue I want to get behind is to make windows executables. If I can't make those, I might as well move to Python - but my programming language preference makes me stubborn about this one. I found out the magical command, ext:saveinitmem. But this alone wouldn't work, and I got to the conclusion SLIME was the problem - the binary image would try to output to slime, and result in an error. The fix is simple enough - just don't load swank, and make the binary without slime. I made a convenience build.bat file, with the following:

c:\dev\clisp\full\lisp.exe -B c:\dev\clisp\full -M c:\dev\clisp\full\lispinit.mem -ansi -norc -q -i build.lisp

(don't forget to change clisp's path to your configuration). One more thing I noted was the dll's clisp uses had to be in the same directory as the produced executable. (find them under "



Then, I put into build.lisp everything I want preloaded. ASDF, my project file loading, etc. and then, the magical function call (assuming an already loaded "test-app" package with a "main" function):

#+:clisp (ext:saveinitmem "test-app"

:init-function #'(lambda ()

(test-app::main))

:NORC t

:script t

:start-package :test-app

:executable t

:quiet t)

All fine so far. The executable is not small, but being about 5MB makes it smaller than the ~30MB I would get with the latest SBCL (1.01) for windows.



Next challenge, the database connection. This was a lot more troublesome than I initially thought. CLSQL was the elected choice, because I needed to access legacy data, initially from an Access .mdb file. I had to create an ODBC Data Source and use the clsql-odbc package to access it. I got this working in the clisp's repl, and then I made an image, run it and BOOM, all hell broke loose. Here is a thread where I got some valuable hints, specially from Ken Tilton, whose Celtk package I used. What happened is that clsql uses CFFI, with cffi-uffi-compat to interface to foreign functions defined in external dll's (in the windows case). Two issues ate more of my time than I would like:

• Each defcfun (i.e., foreign function interface defined) must be evaluated after having used/loaded (only once) the foreign library. Ok, that makes sense. What puzzled me was that, unline Allegro CL, Clisp (and sbcl, while we're at it) requires us to repeat this use/load-foreign-library each time an image is (re)started. This makes it a bit messier, as I have to choose (1) to load clsql at the image execution-time or (2) fetch the needed libraries from the clsql project, and load them in my main (initial) function. I opted from the second approach, for image load-time's sake. The required lines are something like:
  
  

  (cffi-uffi-compat:load-foreign-library "clsql_uffi" :force-load t)
  
  (cffi-uffi-compat:load-foreign-library "odbc32" :force-load t).
  
  

• CLSQL uses a variable to represent a null C pointer. this is ok, except that this pointer is created one time, so after clisp loads the created image, this pointer is invalid, yielding errors every time they are used. Solution? I've simply replaced all *null-ptr* with (make-null-pointer :byte) and *null-handle-ptr* with (make-null-pointer :void). This made the trick, although incurring in a small performancing penalty - I'm not that desperate, yet :).

After this issues, I was able to read and write to my access database from my (now 6.5MB) executable. On to the next issue, the GUI. Ahh, this made me think of lots of variables:

• Language integration with the graphics (consider making calls for foreign functions vs. using simple abstractions in lisp);
• Portability, I wouldn't like to build different GUI's for Mac and Windows, for instance;
• Prettiness. Ok, It may be the best toolkit ever, but I may like to have, say, the xp look and feel. Or some other...
• GUI design. There are some nice GUI designers out there, and there are some nice lisp abstrations to make the same result. What to choose?
  
  

Given this pre-requirements, my options for free were wxCL, graphic-forms, celtk, Ltk, and cells-gtk. (This list may not be comprehensive, feel free to comment, tell me about the amazing />

I tried wxCL first, as I'd already used wxPython, and felt at ease with wxGlade. So I started using it, created my first hello-world widget, and bang, had a whole lot of troubles, cffi related and others. I guess that they could be handled, but I would still be attached to the wxWidgets theme, not very handsome to me. For the same reason I ditched cells-gtk... I moved along, on to Graphic-forms. This is still in alpha stage, but is looking nice! Although, the interface is only for windows forms, and that makes it pretty useless in a mac/linux box. On I went, to the tcl/tk world. At first I thought I'd be attached to a specific theme/motif, but I've learned about Tile, and this made the trick for me. So, how do I choose between Ltk and celtk? Well, Ken Tilton's Cells project, basically, prodded me towards celtk :) It's just a programming language decision, cells can be used to bind behaviors to certain parts of the GUI in an oh-so-elegant way, I was hooked soon enough!



Ok, so what's the downside, you may ask? Celtk is still a work in progress, and is kind of not-ready for quick use/deployment. I also had cffi-related problems, simillar to those I had with clsql, and I worked around them with the following addition to may main function:

(cffi::use-foreign-library "/dev/Tcl/bin/tcl85.dll")

(cffi::use-foreign-library "/dev/Tcl/bin/tk85.dll")

(cffi::use-foreign-library "/dev/Tcl/lib/tile0.7.8/tile078.dll")

There's also lots of absolute paths to fix, in a decentralized way, the asdf is not up-to-date (Ken relies on ACL's *.lpr files), there's no readme/howto/start-here, and basically no homepage with this information. I think Ken is trying to get the project a home in http://common-lisp.net/, and meanwhile the comp.lang.lisp newsgroup will do to discuss issues we may have.



What I can say is that I've put his demo project, along with a sample MSAccess data access, in an ~8MB executable. I'm sure I can optimize this, but for me it's fair enough, and it loads fast enough for me!



Hope some of you may learn something from this. Feel free to shoot any questions, or suggestions you may have!

powered by performancing firefox

On thread persistence and migration

Having contacted the Hop authors, I got a good feeling on their progresses, and I'm looking forward for the next (beta?) version. Also, they suggested me a PhD thesis, "A Portable Mechanism for Thread Persistence and Migration", whose details can be found in Wei Tao's Ph.D. Research Page.



Her dissertation is a good explanation on how a Java program may be adapted into having support for thread suspension and resume. She based her work on a complete code re-writing (both at source and byte-code level), following a (theoretically correct) set of transformations. In short, a program must emulate the creation of a continuation on each code sentence, using an exception mechanism to capture the state of the running program. This exception usage is handy, because after catching an exception the code may throw it again, or store (persistently) the execution state. Obviously there's the problem of non serializable resources having to be saved and restored, and the local, temporary variables. Tao, cleverly leaves this issue to the application creator, with two resolution strategies:

• Creating a wrapper class to the resource, that knows how to close and recreate it (e.g., close a file port and re-opening it a few days later);
• Setting the an existing object field to the resource placed in a local variable;

It's relevant to note that Tao relies on the use of the transient modifier, that makes such a field null when the object is serialized - having to be re-created on deserialization.



The process is heavily focused on the Java Virtual Machine, and the transformation is not proved to make such sense in other programming languages/environments. Nevertheless, the byte-code transformation strategy is well documented and several performance considerations are made throughout the dissertation. I, however, believe that this approach may be simplified - particularly the exception-based technique used to save/restore execution state - when the application is written in some other languages, like Scheme, that natively support first-class continuations.



Her motivation to this technique is well founded, focusing both portability (she may employ this transformations on several applications, and even with other programming languages, as long as their core is closely related to Java's) and code mobility (inevitably the best reason to save thread execution state and to be able to restore/resume it later, although "Moving long running applications from desktops to laptops and back also has clear appeal").



One interesting point she states from the beginning of the report is that making a thread persistently saved is quite similar to having it migrated to another place, the difference being that the store is the run-time environment of another process/machine.



If you're interested in execution-flow migration, thread state, and similar issues, I advise you to read this report, or at least to skim through it.

Mosquito Lisp

I stumbled upon Mosquito Lisp, and thought for a minute that it could be the adaptation to Termite's concepts into a more web-oriented strategy. But I was wrong, as it is only more focused on security measures, and leaves the main attraction in Termite behind - not only it does not allow the exchange of continuations, as it doesn't even consider them in the language (as with macros, now that we're at it!). This can make the dialect extremely simple to grasp and employ on real world cases, but it fails to make academic experiments with abstractions and other language tricks possible (at least throughout several machines!). Here's the link to the introduction:

Ephemeral Security - Introduction to Mosquito Lisp

Application interaction models

I have assembled the interaction types I want to make possible in the near future. They assume a similar architecture to the one verified in current Web Applications, that is, a client-server architecture, a standard (mainstream) browser running on the client and an application container of your choice running on the server. But, unlike today's regular usage of this architecture, I want to allow some other application interaction models:

• Client-Server code mobility, where the server may detect it is unable to compute every requested operation, and sends some of the (possibly still running) computations to the client for it to finish them. As I depict in the preceding diagram, after the client knows the code to the computation he needs to finish, it can gain the ability to perform it without accessing the server (unless there is some server-side information that needs to be fetched, obviously).
  
  
• Disconnected Execution. This sort of interaction is interesting as the client is able to work while being offline (or at least, disconnected from the server). The only requirement is that all of the functionality (perhaps organized in functional modules) and the related domain data must be fetched from the server: As one can see, this may pose other challenges, such as domain data access control (e.g., I may not be able to have full control to some objects that were being used by the server to produce the interface, and thus I cannot have that (whole) object on the client side).
• Server replication, to grant reliability in the provided application. While there are several approaches to do server replication, all of them must allow messages that arrive to a front-of-the-line server to be replicated to all others (or at least the effects of processing the received message). This can be done by the exchange of continuations (see the Termite approach, in my previous posts).
• Inter-client interactions are the most uncommon model, when compared to existing uses of Web Applications. In the "several interactions" arrows can be two main types of communication. There can be only data exchange, like in instant messaging, white-boards or on-line games. But there may also be execution flow exchange. You can have two clients making parallel computations, in a sort of collaborative processing. You can also have ubiquitous execution, like having a smart-phone "pick up" the application execution from the desktop computer's browser and carrying on with it.

While this list may not be complete, it is my starting point. Off course I'll add more examples or complete/modify the existing ones if I see it necessary.



Have a nice weekend!

Mobile computations example

In an attempt to use a new Web diagramming tool (Gliffy), I just put into a simple time-line an example of what I want to make possible with my project:

Basically it is a client that starts by asking the server for a specific code execution. the server, however, opts to stop the execution midway, and transfers its work in progress to the client. Now a couple of noteworthy behaviors can be verified on the client:

1. The client finishes the remainder of the computation, not contacting the server anymore in the process;
2. The client now knows how to make that kind of computations, so it will never bug the server again for that.

I'll welcome any comment on this idea, specially constructive criticism :)