I'll keep this short. I'm tired of workarounds to post what I want the way I want in this blog system. So while this is going to be kept alive on Blogger for reference purposes, I'm going to start using Posterous to publish my random thoughts, academic/professional facts and/or opinions. Thank you for having read along. If you have been enjoying the content, please head on to http://edgargoncalves.posterous.com, and update your feeds to that blog's.
Friday, July 02, 2010
Friday, June 18, 2010
How I live with cold starts on GAE's JVM
It has come to my attention the (obvious and logical) fact that loading the JVM is expensive, time-wise. And Google's Application Engine (GAE) makes no exception, so whenever one servlet is "hit" long after any previous server interaction, a JVM is started. In practice, this means that we're looking at a "loading..." spinning wheel with a duration of about 10s, +-5s. That's kind of a bother, but that's not the worse. If we're talking about a single-page modeled web app, the user is bound to leave it opened for some time (minutes, not to say hours, before he is able to gather all the info to fill in that devilish form!). And after that time, when he is eager to finish up his task (that has already took him enough time outside the browser window), guess what? That's right, wait another 10s for the server's JVM to wake up. Off course this is a non-issue for heavy-traffic web apps, but this platform is great for experiments and small business products, and it's this niche I'm worried with (perhaps because I'm also targeting it :) )
Today's scenario is yet a bit worse. The time it takes for a GAE JVM to get drowsy is often shorter than 5 minutes (here are some conclusions about timings, as well as other suggestions to alleviate the problem - but insufficient, still). I could only imagine Google is having a hard time escalating their servers to everyone, but they should get a solution to this. Otherwise people will start hacking this problem subversively. In fact, they're already doing it: exploits are there to make a repeated task that keeps the JVM in question alive - a ping, let's call it that! But as soon as this hack becomes mainstream and widespread, Google will hog down with requests. So I'm sort of against it. But since there is no viable solution so far, I started reconsidering things.
What I ended up was a compromise. For applications like the ones I'm developing there will be only 3 or 4 users, and their request will be very scarce and scattered. So I will not use a frequent and constant ping bot. Instead, I made a client-side that does that pinging, but only while the application is opened. So yes, the user has to wait some seconds to fire up the application for the first time. But afterwards, a background (i.e., asynchronous) periodic request keeps the JVM ready to listen for the application's demands.
Here's a possible Javascript code to make this happen. The trick here is to create a script dynamically (i.e., appending it to the body), and setting its source to any servlet's served URL (in this case, a ping one).
Monday, June 07, 2010
International Lisp Conference 2010
I was asked to spread the word about the upcoming International Lisp Conference 2010 (ILC-2010). So here you have the Call for Arms Papers (from the original attachment):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
International Lisp Conference 2010
October 19-21, 2010
John Ascuaga's Nugget (Casino)
Reno/Sparks, Nevada, USA (near Lake Tahoe)
Collocated with SPLASH 2010 (OOPSLA & DLS & more)
see also http://splashcon.org as well as
http://www.dynamic-languages-symposium.org/dls-10/
In association with ACM SIGPLAN (PENDING)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Association of Lisp Users is pleased to announce that the 2010
International Lisp Conference will be held in Reno, Nevada, in
collocation with SPLASH 2010. The scope includes all areas related to
the Lisp family of programming languages.
Accepted papers will be published in the ACM Digital Library (PENDING).
Extended Abstracts and Papers must be written in English and submitted
electronically at http://www.easychair.org/conferences?conf=ilc2010 in
PDF or WORD format. If an Extended Abstract is submitted, it must be
between 2 and 4 pages, with full paper to follow before final deadline.
Final submissions must not exceed 15 pages and need to use the ACM
format, for which templates which can be found at:
http://www.acm.org/sigs/pubs/proceed/template.html.
Important Dates:
~~~~~~~~~~~~~~~~
* Deadline for Abstract Submission: August 1, 2010
* Deadline for Paper Submission: September 6, 2010
* Author notification: September 20, 2010
* Final paper due (in electronic form): October 5, 2010
* Conference: October 19-21, 2010
Scope:
~~~~~~
Lisp is one of the greatest ideas from computer science and a major
influence for almost all programming languages and for all
sufficiently complex software applications.
The International Lisp Conference is a forum for the discussion of
Lisp and, in particular, the design, implementation and application of
any of the Lisp dialects. We encourage everyone interested in Lisp to
participate.
We invite high quality submissions in all areas involving Lisp
dialects and any other languages in the Lisp family, including, but
not limited to, ACL2, AutoLisp, Clojure, Common Lisp, ECMAScript,
Dylan, Emacs Lisp, ISLISP, Racket, Scheme, etc.
Topics may include any and all combinations of Lisp and:
* Language design and implementation
* Language critique
* Language integration, inter-operation and deployment
* Applications (especially commercial)
* 'Pearls' (of wisdom)
* Experience reports and case studies
* Reflection, meta-object protocols, meta-programming
* Domain-specific languages
* Programming paradigms and environments
* Parallel and distributed computing
* Software evolution
* Theorem proving
* Scientific computing
* Data mining
* Semantic web
We also encourage submissions about known ideas as long as they are
presented in a new setting and/or in a highly elegant way.
Authors concerned about the appropriateness of a topic may communicate
by electronic mail with the program chair prior to submission.
Each paper should explain its contributions in both general and
technical terms, identifying what has been accomplished, explaining
why it is significant, and comparing it with previous work. Authors
should strive to make their papers understandable to a broad audience.
Each paper will be judged according to its significance, novelty,
correctness, clarity, and elegance.
The official language of the conference is English. Some further
information is available at the conference web site, with more details
added later. See: http://www.international-lisp-conference.org
Technical Program:
~~~~~~~~~~~~~~~~~~
Original submissions in all areas related to the conference themes are
invited for the following categories.
* Papers: Technical papers of up to 15 pages that describe original
results or explain known ideas in new and elegant ways, or extended
abstracts of 4 pages soon followed by the corresponding full paper.
* Demonstrations: Abstracts of up to 4 pages for demonstrations of
tools, libraries, and applications.
* Tutorials: Abstracts of up to 4 pages for in-depth presentations
about topics of special interest for at least 90 minutes and up to
180 minutes.
* Workshops: Abstracts of up to 4 pages for groups of people who
intend to work on a focused topic for half a day.
* Panel discussions: Abstracts of up to 4 pages for discussions about
current themes. Panel discussion proposals must mention panel
members who are willing to partake in a discussion.
* Lightning talks: Abstracts of up to one page for talks to last for
no more than 5 minutes.
Depending on the technical content, each submitted paper will be
classified by the program committee as either a technical paper or as
an experience paper; and authors will be informed about this
classification. Note that all interesting submissions are considered
valuable contributions to the success of the ILC series of
conferences. As in past ILC's since 2007, accepted papers in both
categories will be presented at the conference, included in the
proceedings, and submitted to the ACM digital library.
Organizing Committee:
~~~~~~~~~~~~~~~~~~~~~
* General Chair:
JonL White - The Ginger IceCream Factory of Palo Alto, ALU
* Program Chair:
Antonio Leitao - Instituto Superior Tecnico/INESC-ID
* Conference Treasurer:
Duane Rettig - Franz, Inc., ALU Director
* Publicity Chair:
Daniel Herring - ALU Director
* ALU Treasurer:
Rusty Johnson - TASC, Inc., ALU Director
Program Committee:
~~~~~~~~~~~~~~~~~~
* Antonio Leitao - Instituto Superior Tecnico/INESC-ID, Portugal
* Alex Fukunaga - University of Tokyo, Japan
* Charlotte Herzeel - Vrije Universiteit Brussel, Belgium
* Christophe Rhodes - Goldsmiths College, University of London, UK
* Didier Verna - EPITA Research and Development Laboratory, France
* Duane Rettig - Franz, Inc., USA
* Giuseppe Attardi - University of Pisa, Italy
* Jeff Shrager - Symbolic Systems Program, Stanford University, USA
* Joe Marshall - Google, Inc., USA
* Julian Padget - University of Bath, UK
* Keith Corbett - Clozure Associates, USA
* Kent Pitman - PTC, USA
* Manuel Serrano - INRIA Sophia Antipolis, France
* Marc Feeley - University of Montreal, Canada
* Marie Beurton-Aimar University of Bordeaux 1, France
* Mark Stickel - SRI International, USA
* Matthias Felleisen - Northeastern University, USA
* Scott McKay - ITA Software, USA
Contacts:
~~~~~~~~~
* Questions: ilc10-organizing-committee at alu.org
* Program Chair: ilc2010 at easychair.org
For more information, see http://www.international-lisp-conference.org
I'm curious about when are "clojurers" going to join forces with the rest of lispers on these kind of conferences. Specially since this one is collocated with SPLASH 2010, maybe it isn't such a bad idea to start doing it now :)
Monday, May 31, 2010
Yet another Clojure + Compojure + Google App Engine post
Have I mentioned I'm kinda late to this show? Well, better late than sorry. Anyway, I think you can't have too much of these, specially in times where google application engine (GAE) is growing and so is Clojure.
The scenario was like most others: I wanted to build my webapp using GAE and clojure, and also stay on the bleeding edge of each technology. So lets see what I've got. I started by taking a look at the "literature" available as blog posts and group discussions (such as this and this).
I installed lein, and created a standard project.clj (By force of habit, forgot to use "lein new", but it's practically the same). I added the following directives on defproject:
:source-path "src" :compile-path "war/WEB-INF/classes/" :library-path "war/WEB-INF/lib/" ;; namespaces will trigger AOT compilation to bytecode. you want this! :namespaces [ginasiosunderground.servlet]My goal is to produce a war directory, to upload to gae servers. So I told lein to compile to war/WEB-INF/classes, using war/WEB-INF/lib as a classpath dir. (More to come later on that :namespaces directory). The final project.clj is the following:
You can see the list of dependencies. I'm using the latest clojure 1.2, compojure 0.4, hiccup 0.2.4, a ring 0.2 (specifically adapted to use with gae servers) and appengine-clj (from r0man's fork). I also use clojure-json (to make my life easier on jQuery client side ajax calls). I'd like to mention the libraries from google. As of now, a default lein setup can't find those up to date versions (they are available, but I didn't want to toy around with maven repositories to make the magic happen). So I got those jars from the gae java sdk (download it from gae's site) and placed them on my war/WEB-INF/lib folder. Now be ware, you should have a copy of them somewhere else, because each lein clean you do will delete them from the lib directory! (eventually I'll get around to automate this process. I also know there's a "lein gae" plugin somewhere, but haven't tried yet, I'm not sure if it does what I want). One last thing about lein, you'll have to remove commons-fileupload-1.2.1.jar from your lib dir, otherwise gae will complain about it (it did, for me!).
Let's move on, then. my-app.servlet loads some stuff I need for the servlet, generates an HttpServlet class, and imports some classes so that I may load a jetty server and use gae's services (datastore, etc) in interactive mode (read, using SLIME with swank-clojure). Here's the namespace:
The magic that's left is, after issuing the defroutes and defservice command, the following (thanks to r0man for the tip!):
So here we are, running a webapp that can be deployed to gae (if you have read the linked posts, you'll see that you also need a web.xml and a appengine-web.xml to deploy, and you'll also need to comment out the ring.adapter.jetty from your namespace - gae already has its own application server :). Now by running the local server via (start-server) and not via gae's dev_appserver.sh shell script, you can happily work, on slime, redefine your servlet actions and refresh the browser to see your changes. Yay!
One problem, though. How do I use google's services? appengine-clj to the rescue, browse the code (see the test-suite for examples, specially on how you can test them locally using macros like appengine-clj.users/with-user-info). I was basically a happy camper until I got to try out user logins. GAE has this method, createLoginURL and createLogoutURL (applied to a UserService and a destination page string), that takes us to a dummy login screen. To be able to use this Service, I wrote the following code (loaded before starting the jetty server!):
But while this compiles perfectly, I noted that the login screen is not loaded, i.e., clicking the login link yields a 404 page. Not good. But it should be loadable, since the dev_server has it. Google's documentation isn't much help here, and I can't seem to find how on earth I access that screen (I need it to try out administration pages!). So feel free to drop a comment if you happen to know the solution for this!
To conclude, I'm not going to publish my application as a "blank example", as there are already some around. That said, if you have a question about this setup, just ask and I'll be as helpful as I can!
Friday, May 28, 2010
On Clojure error messages, II: ask and ye shall receive
I now realize my previous post carried some strong emotions, most of them not too good, and I want to humbly apologize for it. Let me state (again) that I'm enjoying working with clojure (for the most part, at least).
I wrote about the useless (wrong?) error messages, specifically when passing a list instead of a vector (ISeq) as arguments for a defn. After experimenting a bit, I was even more confused: (defn x () 1) complaints about the integer, so does (defn x () [] 1) and (defn x () [1]), clojure just doesn't know how to create an ISeq from that integer. I now understand this makes sense because you can specify multiple arity functions, like this:
(defn x
([] (x 0)
([x] (+ x 1)))
So in this case, if a vector isn't found where the arguments are expected, clojure complains, as he can't make an ISeq.
But none of this matters anymore (to the language user). Stuart Halloway (author of the book that I used to learn about Programming Clojure, which by the way I advise you to read, if you haven't already) was almost to fast picking up this issue! I have just received his tweet with a fix to this problem: clojure (the snapshot version) now detects if the arguments are a proper vector, and if not warns us with a much clearer message: "Parameter declaration arg1 should be a vector". This should satisfy all with the same issue I tackled on the previous post; more advanced users dealing with multiple arity defns should already be aware of the syntax, so it's safe for both cases.
This was a positive outcome in more than one way. Not only clojure got a bit better, but I got reminded of how I should had followed a much nicer "netiquette". I made a public complaint, and Stuart was impeccable to pick it up, hacking a fix and letting me know about it (everyone should learn from him, including myself!). But instead of posting my complaint (again, sorry if my tone was to harsh), I should had (1) ask on a discussion list or on other social platform about this; (2) take a look into the source code, see where the issue could be fixed, write a patch, and submit it to approval to the developers - alternatively, take the oportunity to learn how to do those things, and at least let the maintainers know about it, so a bug report is issued. This is the organization that most projects require to evolve properly.
Thursday, May 27, 2010
On Clojure error messages.
I know. I'm late to this train. And there are lots of resources to clarify this. But I have to manifest myself somewhere (and twitter's char limit won't do this time).
Here's the deal. I know I've written a fair share of Clojure lines of code. But I come from the land of the ancient parenthesis (specifically, Scheme, Common Lisp). So whenever I try to teach clojure to someone, it's inevitable to tell them that it's a Lisp, so it's ok to see those rounded delimiters everywhere.
This is where I start by asking them to define a simple function, and instead of defun/define, use defn. Something like this will do:
(defn xpto ()
(+ 2 3))
Simple enough, right? Wrong. (I know it's wrong, now, off course). But for the beginner, the first thing to do is to read the error message. So let's take a look:
Don't know how to create ISeq from: clojure.lang.Symbol
[Thrown class java.lang.IllegalArgumentException]
Yeeeah, isn't that obvious? Clojure wants to make a sequence and I'm trying to use a symb... wait, what?? the only symbol I know I'm using is xpto, and that's about right on the syntax! The Java exception isn't much help either. What argument? The ones I'm not using on the functions (but weren't those parameters?) Or the ones I'm using on the function call (but 2 and 3 are numbers, not strings!! *help*!) Ok, breath. let's take a look into the stack trace, it is usually helpful on java, at least to trace where the error is:
Backtrace:
0: clojure.lang.RT.seqFrom(RT.java:469)
1: clojure.lang.RT.seq(RT.java:450)
2: clojure.lang.RT.first(RT.java:538)
3: clojure.core$first.invoke(core.clj:53)
4: clojure.core$sigs$asig__3331.invoke(core.clj:202)
5: clojure.core$sigs.invoke(core.clj:214)
6: clojure.core$defn.doInvoke(core.clj:268)
7: clojure.lang.RestFn.invoke(RestFn.java:495)
8: clojure.lang.Var.invoke(Var.java:381)
9: clojure.lang.AFn.applyToHelper(AFn.java:180)
10: clojure.lang.Var.applyTo(Var.java:482)
11: clojure.lang.Compiler.macroexpand1(Compiler.java:5215)
12: clojure.lang.Compiler.macroexpand(Compiler.java:5270)
13: clojure.lang.Compiler.eval(Compiler.java:5338)
14: clojure.lang.Compiler.eval(Compiler.java:5320)
15: clojure.core$eval.invoke(core.clj:2366)
16: swank.commands.basic$eval_region.invoke(basic.clj:47)
17: swank.commands.basic$eval_region.invoke(basic.clj:37)
18: swank.commands.basic$eval801$interactive_eval__802.invoke(basic.clj:66)
19: clojure.lang.Var.invoke(Var.java:365)
20: myproject.core$eval3896.invoke(NO_SOURCE_FILE)
21: clojure.lang.Compiler.eval(Compiler.java:5353)
22: clojure.lang.Compiler.eval(Compiler.java:5320)
23: clojure.core$eval.invoke(core.clj:2366)
24: swank.core$eval_in_emacs_package.invoke(core.clj:94)
25: swank.core$eval_for_emacs.invoke(core.clj:241)
26: clojure.lang.Var.invoke(Var.java:373)
27: clojure.lang.AFn.applyToHelper(AFn.java:169)
28: clojure.lang.Var.applyTo(Var.java:482)
29: clojure.core$apply.invoke(core.clj:536)
30: swank.core$eval_from_control.invoke(core.clj:101)
31: swank.core$spawn_worker_thread$fn__453$fn__454.invoke(core.clj:300)
32: clojure.lang.AFn.applyToHelper(AFn.java:159)
33: clojure.lang.AFn.applyTo(AFn.java:151)
34: clojure.core$apply.invoke(core.clj:536)
35: swank.core$spawn_worker_thread$fn__453.doInvoke(core.clj:296)
36: clojure.lang.RestFn.invoke(RestFn.java:398)
37: clojure.lang.AFn.run(AFn.java:24)
38: java.lang.Thread.run(Thread.java:637)
Read it all? Want your 3 minutes back? (ok, 30 if you actually went looking for documentation on the API, the source of Compiler, core.clj or worse if you went looking for slime's or Thread :) ). Sick, isn't it? not a single line is useful for our cause.
I'm afraid I have no good news for you, except to tell you what the problem was and hope you'll never forget that. I know I didn't. (until I swapped to a bunch of different programming languages, including common lisp, for a while, and returned with my auto-pilot coding style turned on :) ).
The problem is that clojure expects a sequence of parameters, not a list. And thus, the proper syntax is to use [] instead of (). I know, a list is not a symbol. Or so I thought... I could even go theorize about () being translated to nil in compile time, and so it was a symbol. But evaluating (symbol? nil) quickly clarifies that for us...
Pledge - please, oh please start adding useful semantic to clojure's core language errors... The language entry barrier will lower significantly, I'm sure!
Thursday, August 20, 2009
Aperture: scripting takes you where automator doesn't
Like some unfortunate people, I've suffered from data loss. Mainly by distraction and a trigger-happy finger on the delete key. So I've embraced backup strategies, and my last acquisition was a Western Digital MyBook World Edition with 1TB hard disk inside. I keep it connected to my router, so it's always available from a wireless laptop in the house. I'm happy and comfortable with the setup, so the next step is, no surprise, figure out what I'm going to put there. And, most importantly, how I'm going to do that.
My first use case was my photography collection. I've recently started using Apple's Aperture to handle my pretty basic photography sets. I don't own a SLR, yet, but I have a Canon Powershot S50, which allows to manually configure the machine to take some interesting photos. And it shoots RAWs (albeit in a Canon proprietary CRW format, that can be batch-translated to DNG using the freeware converter from Adobe). So after my photo workflow I end up with an hierarchy with folders (for project types like "Trips and Events", "Experimentations", "Personal", "Work", etc), inside which i have projects.
What I wanted to achieve was a threefold backup: Save the project itself (so that i may open it in other aperture libraries, both on my macbook pro and my desktop mac); Save the masters, raw files; Save the versions, in JPEG. I figured I'd try Apple's little Marvin, the Automator, to help me easily do it. I got excited to see that there were some export actions available for use in workflows! That, however, proved to be insufficient, as there is no action to get the currently selected project. This was the first applescript i used on the workflow, to get just that, and place in a variable to pass on later. The code looks something like this:
tell application "Aperture"
activate
-- identify currently selected image(s) project
set x to selection
set x to item 1 of x
tell library 1
set ap_proj to (get value of other tag "MasterProject" of x)
end tell
end tell
I ended up giving up the entire workflow, and scripted the whole thing. I only kept the automator workflow file so that i could easily add stuff like growl warnings. You can download it from here and give it a try. Oh, and if you know an easy way to find out the folder a project is in, to put the bundle in that directory, please, leave a comment or tweet me!
With this script (with a small adaptation) it's also easy for me to loop through all projects and perform a complete backup of the library. This sure beats the workflow-actions way, as there's no amount of variables and loops that can make you run through a list of projects. For these reasons, I believe Automator needs a 3.0 update. Maybe Marvin could take some anti-depressives, and leave the paranoia behind? :)
Subscribe to:
Posts (Atom)
