Sunday, March 23, 2008

All those parenthesis are counting on you

Let's make something clear, everyone designing their own dialect of lisp should be solving real problems instead. Still people continue to do that.
Because :
a It's fun
b It looks easy.
So what every lisper ought to do before he dies:
"Plant a tree, write a book, create his own dialect of lisp"

It's not that there is no benefit in making a better lisp, but the problem is that core language is only a part of the equation, the rest of the parameters in the formula are implementations, libraries, community and learning material. There are probably more but those will do. Also, the core language of lisp is well designed, showstoppers are few and controversial (*), so you're left with polishing language quirks that could be done within the language using metaprogramming facilities. Suddenly your new dialect of lisp is only a little bit better than language you're trying to supersede, but guess which one has more libraries, implementations, books, tutorials and users. Your chances aren't looking good now. You could design a language with different trade-offs and win some *markets* but don't attack a fortified hill. You'll need a lot of firepower to stand a chance.

Having a spare time and lacking a computer, left my old pc at home and still deciding what kind of notebook to buy, I started thinking about polishing some quirks and integrating some features I like from other languages. And the thing I hate most in lisp is the compound loop macro.
Why? It's such a useful thing. And why don't you complain about format too?
I don't use format that much so even if someone fix it, that won't buy me a lot. Something like if black pepper price doubles it won't hurt your budget because it's only 0.0002 % of your income but imagine paying twice for rent or gasoline. Ouch, that hurts.Because whatever I write it involves some kind of looping. Sometimes you can get away with higher order functions, sometimes dolist & dotimes will do, but more often than I want, loop is the best, read succint and efficient, solution. You can beat(**) loop on succinctness or effectiveness, but it's very hard or impossible to beat loop on both. Loop macro is special kind of demon, a DSL. Generic tools does very little damage on it.
Take this code for example source:

(defun separate (lst)
(mapcar #'(lambda (n) (mapcar #'abs (remove-if-not n lst)))
(list #'plusp #'zerop #'minusp)))

(defun separate (list)
(loop for element in list
if (plusp element)
collect element into positives
else if (minusp element)
collect (abs element) into minuses
else collect element into zeroes
finally (return (list minuses zeroes positives))))

My code is far shorter than Rob St Amants' but guess whose the consing whore (***). So I'm struggling to find a replacement to loop. Yes I've tried iterate, and it's better than loop but it's not that better. Like somebody described it : putting few extra parenthesis doesn't make iterate much lispier than loop. Also I've tried series and it's just not it. You can't argue with taste.

So what's the current situation in my mind. I have a good idea how to get rid of many stinking loops, that will require uniform access to lisp containers, which is good because that's another quirk that could be polished, also I have an idea how to simplify anonymous function generation in special cases. The problem is what to do with loops that collect multiple values like the code above. Using let is like rising the white flag of defeat, but I still don't have a graceful idea how to solve the problem. Could anybody implement above loop efficiently with higher order functions and without let? I'm really hungry for inspiration.
I'm thinking about a graceful solution, but good design is hard to get right. Unifying concepts of initializing, stepping, modifying and returning multiple results looks like trying to make a crossing of race car, jeep and truck and make sure that it doesn't look or behave like Frankensteins' monster.

(*) Continuations vs unwind-protect, better reflective abilities against cost of implementing those, etc
(**) Equal is fine also.
(***) Paraphrasing Ken Tilton.

Saturday, March 22, 2008

Real lispers

Real lispers don't write programs , they write programs that write programs.
Real lispers don't solve their problems, they shape lisp into something that will solve their problems.
Real lispers know a lot of programming languages, they invent one in every program they write.
Real lispers use common lisp, scheme is for head-in-the-clouds academics, emacs is just a typo.It should be zmacs or climacs.
Real lispers don't write dialects of lisp, that's for failed wannabes that can not grok common lisp.
Real lispers don't need paren matcher, they haven't seen a single paren from the day they became masters.
Real lispers code in lisp machine, everything else is just Allegro prompt with some pretty colors.
Real lispers use MIT or BSD license, if you're good enough to use it, you can have it.
Real lispers make libraries, wannabe assemble hackbraries.
Real lispers code is like poem, wannabe code is just a bunch of typing, mispelled words and overuse of parens.
Real lispers documentation is like a novel, helpful and great read. Wannabe documentation is like an shabby phone book, filthy and outdated.
Real lispers don't need FFI, they rewrite the foreign part in lisp.
Real lispers don't use loop, loops is someones poorly written DSL that somehow slipped into the standard.
Real lispers don't use format, obfuscation contests are boring.
Real lispers might not know their wife name , but they know all the ANSI Common Lisp standard including the cleanup issues.
Real lispers snubs APIs, APIs are for tell-me-where-to-pour-the-concrete programmers.
Real lispers don't whine about lack of libraries, they write everything they need.
Real lispers introduce features, wannabes introduce bugs.
Real lispers code works across OSes and implementations, wannabe code is 99.99% finished for years.
Real lispers love lists.
Real lispers use very-long-and-descriptive-hyphenated-names.

Thursday, March 20, 2008

Lisp is a river

Lisp is a river, quick stream of water.
running with fast flow
Take your kayak and enter the current
toward the rapids, away from shore

Lisp is a river, wide stream of water
running with mighty flow
Drifting on surface, looking straight forward
you are a captain of a riverboat

Lisp is river, endless stream of water
flooding your shores
But you are a shaman, you’ll raise new land
no one could hurt you, you're in control

Tuesday, March 18, 2008


Yesterday I got a friendly email suggesting that I should stop learning those esoteric languages and switch to Arc(*), presto. After the usual propaganda, that Arc is the latest, greatest and streight from the roots of original McCarthy's lisp blah blah blah, I was finally shown an example of such ingenuity with its if operator. Spice that with comments like what the hell was common lisp comittee thinking when they buried cond under sea of parenthesis and you'll know why the blackhat character from xkcd has zero tolerance for newbishness. So why the hell Arcs' if is not such a big deal?Anybody who spent more than 15 minutes reading On Lisp will find writing such macro nobrainer. Here's mine code as you go:

(defun pair-them (x)
(cond ((null x) nil)
((endp (cdr x))
(cons (list t (car x))
(pair-them (cddr x))))
(cons (list (car x) (cadr x))
(pair-them (cddr x))))))
;;; Grahams' if
(defmacro gif (&rest args)
`(cond ,@(pair-them args)))

Few tests:

'(gif (= a 1) (setq a 2)
(= a 2) (setq a 3)
(= a a) (floor a 3))))

(COND ((= A 1) (SETQ A 2))
((= A 2) (SETQ A 3))
((= A A) (FLOOR A 3)))

'(gif (= a 1) (setq a 2)
(= a 2) (setq a 3)
(floor a 3))))

(COND ((= A 1) (SETQ A 2))
((= A 2) (SETQ A 3))
(T (FLOOR A 3)))

Looks fine. I could probably make something nicer but this is Ok for demonstrational purposes. Note that if you feed it with odd number of arguments it assumes that last clause is the default one. I don't know exactly how Arcs' if is working, but I assume it's something like that.
But what will happen if you have a cond clause with multiple computations like below?

(cond ((evenp x)
((oddp ...

Than you'll have to insert some progns and the beauty is gone.

(if (evenp x)
(do-something-else)) .
(oddp ...

So you pay for those omitted parens with additional progns. Whichever solutions is better depends from the situation, so use whatever you want. If someone could think of graceful solution that could simoultaneously omit extra parens and progns that could make me learn Arc. Or rather implement the same thing in common lisp.

(*) I didn't even looked at Arc,beside it's cover.

Wednesday, March 12, 2008

Farewell Haskell

UPDATE 05/08/2014
After writing a lot of functional code in JavaScript using LoDash I'm starting to appreciate the type system.

I decided to stop torturing myself and part with Haskell. Why I did that?1st I don't give a damn about its type system and Haskell was putting it in my throat.2nd I didn't learned anything new. Well beside bunch of tricks, and I wasn't looking for tricks. I expected new paradigm, novel idea or an elegant combination of existing ones but there was nothing of those to inspire me to continue wrestling with the static typing.The best description of static typing came from Aatu Koskensilta
One of the reasons to prefer a powerful type system is precisely the ability to encode some such conditions and information in the types. The usefulness of a type system depends on which conditions are expressible in it, and with what amount of effort.
In any case, if one does not use the type system for anything useful it obviously will be nothing but a burden. Learning to program in languages such as Haskell involves learning how to use its type system to do useful thing, just as one would learn to use macros etc. when learning Common Lisp or Scheme. Don't think of the type system as a straight-jacket, think of it as something to hack, a tool -- or just continue writing your code in a dynamically typed language if that's your thing.

And static typing is definitely not my thing. But if I decide to get in those waters someday I will look no further than Qi , it has the most powerful type system, it's optional and I could even hope that it'll be asdf packaged. My current state of mind is probably best described by below :
It may interest you to know that most of my development time for that was spent typing: paging through the library, trying things out at the REPL, tweaking, debugging, iterating. It's a style I find much easier than staring at a blank screen and thinking very hard, and one for which I find forgiving languages like Perl and Lisp are much better than Haskell. - Ladislav

It's good to have a heroes, but sometimes it's even better to see them crucified, just to make sure they're worthy.

Farewell Haskell, you're joining the company of languages that didn't taught me much about programming nor changed the way I'm thinking .As Alan Perlis once said: "A language that doesn't affect the way you think about programming, is not worth knowing". The rest of the crew consists of pascal, basic, delphi, php, c#, ocaml, python and sql.

Saturday, March 8, 2008

Lisper's first look at Haskell

I started learning Haskell a week ago and after few days I already wrote a lengthy essay about Haskell, and most of it wasn't singing praises to it. Quite the contrary, it was how it sucks. The reasons vary but it probably has something with lispers' being spoiled brats. They're damn hard to impress. You show them the cool feature that your favourite language has and they either already have it, have something better or they've already seen it and decided it's not such a good idea as you might think. So this essay is from lisper point of view. I doubt that users of other languages will find much use of it.

But first things first. Let me tell you why I decided to learn Haskell.
1st Haskell is purely functional, variables could get assigned once only, so I wanted to know how the hell they deal with keeping state.
2nd Few lispers like Slava and Frank knew Haskell.
3rd Haskells' code is very compact.

So I grabbed Winhugs and went Huskying. I quickly went through YAHT but I was really disappointed. That made me into writing that lengthy Haskell sucks rant, but after little thinking I understood that I was learning Haskell the lispy style, read Gung-Ho. Damn,I was practically trying to write lisp in Haskell and whenever Haskell complained I was furious. So I decided to give it another shot, this time with Haskell The Craft of functional programming. Halfway through the book I decided to clean my mind of my current thoughts about Haskell by writing this article.
So it presents current state of my mind. Keep in mind that I'm a multipass doer, it needs several iterations before I make something good. So I'll start with little philosophical view.

Design of Haskell:

There is a nice article called What is a Ryu? that explains the definition between a classical japanese martial art, koryu and modern ones like judo. I found it very enlighting in comparing computer languages. To save your time from reading it I extracted few paragraphs. So what's difference between ryu, classical martial art and do, modern ones.
The word ryu roughly translates to style or school but the real meaning is "to flow, flowing... system or school," and here's how Wayne explains how the ryu is created.:
The founder of the style experienced what amounted to a divine revelation. These experts had already developed a vast repertoire of technical knowledge through a study of martial methods and through actual experience in battle and/or in training. But having exhausted and reached the limits of their technical expertise, they consciously underwent shugyo, or a rigorous training that tested their mind, body and spirit.

Often enacted in the confines of sacred ground such as a Shinto shrine or Buddhist temple, or in a hidden religious refuge in the wilderness or mountaintop, shugyo was meant to crack through the surface layer of the physical world to lay bare the secrets of the spiritual universe. Like the intensity of Zen training, after exhausting the everyday avenues of awareness, the trainee undergoing martial arts shugyo attains a new and enlightening insight.

After this period of intense training, prayer, and some kind of fasting and/or abstention, a vision would appear to the founder, that would give him the key to true mastery of his art. Often just a simple phrase or very rudimentary technique(s), the revelation would be the key that unlocked all the subsequent methods that the founder would develop. As such, the revelation was tenshin shoden, or knowledge bestowed from the Heavens, or muso; knowledge gained from a heavenly dream.

The knowledge, therefore, was heavenly pure when it was first passed from the gods to the first generation. If it is to remain a divinely inspired system, then it must always flow back to this person, the founder. Hence, the term -ryu, or nagare; it is a flow back to the wellsprings of the style, which was divine inspiration.

Now let's compare ryu with do, meaning the way. the modern craft.

Judo, as a do form (a "modern" type of martial art stressing mind, body and spiritual development over self-defense or combative methods), is just judo, period. You may have a Kodokan style of judo, a Japanese college style, an Olympic style, but because it is so all-encompassing, there are no ryu in judo. Judo is so malleable in techniques, and is so reasonable in its scientific application of contest-oriented grappling methods that it continues to grow and innovate in technical complexity. It is in fact difficult to truly innovate and create something completely "new" under the sun. Most of us martial artists in this day and age simply aren't capable of such stunning originality.

In case you skipped Muromoto description , ryu is something created with divine inspiration, deity called Alonso Church visited John McCarthy and he combined that vision with s-expressions and made something original, something beautiful.

The first lisp. Common lisp, scheme and emacs lisp are just ryuha (factions ) within a lisp ryu. They're part of the lisp flow but they ain't so pure like the founders original idea. That's why Paul Graham tries to get back to the roots of the original lisp with his Arc. Will he succeed that's another question.
From the other side we have do, 'modern' martial arts they "were developed by committees of human beings, whose attempts are in synch with the modernist trend that things can always be improved with time and human ingenuity.". Most of the modern languages are like do martial arts. Those are languages that weren't blessed with heavenly inspiration. It isn't that their authors wasn't smart or they weren't great programmers, quite the contrary, it's just that muse of inspiration didn't visit their door when they were designing their language. So they did what they did with what they had.
Haskell is one of those 'unblessed ', modern do languages sharing company with c++, java, c#, OCaml, F# etc. If you compare Haskell with some ryu language like Prolog, also one I recently started learning, the difference is obvious. When I tried Prolog I said 'Wow', all the language could be made just from rules ( facts are just degenerated form of rules). On the other side we have Haskell with their bulky kernel language and complex rules of composition. Haskell designers took features from older languages Lisp and Prolog and mixed them together with their own ideas. That's nothing wrong with it, humans build on existing foundations. The problem is that they failed to unify those techniques, so instead of making a monolith tool they ended with "a mess of illogical patchwork". It's not that result is unusable, quite the contrary, but it's not a masterpiece either.

After this philosophical introduction let's get little practical. How does programmming Haskell feels.(This list will probably get changed as I'm learning more of the language)

The good :
1. It teaches you a lot about functional programming. Lisp encourages functional programming , but Haskell enforces it. So all your nasty habits you picked up coming from imperative and object oriented languages will have to go away. There is no looping and variables could be assigned only once. Most of the Haskell constructs for functional programming are already in lisp, but lisp allows you to write imperative and OO code if you want. That won' t do in Haskell. Remember about the composing functions , and returning functions as values. Well I didn't used them much in my lisping. Haskell taught me about their value.
2. You got compact code for free. As soon as your c program is working its probably fast. The same goes with Haskell, as soon as your code is working it's short. With lisp you usually have to work to make your code with c-s' speed or haskells' size.
3. You'll learn a lot of functional constructs, that could find a good use in your lisp arsenal, point free programming, curry, uncurry, flip etc.
4. You'll understand the value of laziness and generators. There's a cool library called heresy made by Matthew Lamari. And you might befriend with your old but long forgotten acquaintance series
5. Built in pattern matching - it's very convenient about programming sometimes. Unfortunately for someone who learned Prolog before Haskell, Haskells' pattern matching will look very limited.
First it doesn't allows me to repeat a variable in a pattern , second compared with Prolog, it's one sided. Big issue for someone used to full power of unification.
6. Algebraic data types are neat.

The bad:
1. Static typing sucks. It reminds me of the dark ages of programming in the c family of languages. Inference engine saves me from writing the boring type declarations but it's far from perfect. I have nothing against static typing if it's optional. One of the coolest features of lisp that I found when I started learning it coming from c++ was thatI could put anything into lisp containers. And I don't want going back.

2. No rapid development. Don't get me wrong, developing in Haskell is faster than in c++ or c# but compared with ultimate RAD vehicles like lisp or prolog, haskell looks like a family caravan against full blood racers. Haskell doesn't allows me to experiment. It doesn't allows me to create modulus with some functions undefined or defined wrong. Bundle this together with loading file to redefine a module, the cruppiest feature they took from Prolog and you'll start to value your lisp listeners.

3. Hogs a lot of brain resources. It's syntax is complex, and together with static typing it forces me to spend more resources on dealing with language than with dealing with the problem.

4. Inflexible- Remember the Alan Kay quote - Lisp isn't a language it's a building material. Not so with Haskell. You can not wrap the Haskell around the problem as you can do with Lisp. Haskell forces you to work the other way around.

5. Spartan development enviroment. Compared with lisp offerings WinHugs looks poor man's IDE. There are so much things that are missing that I don't know where to start. Beside I had to write Haskell code in notepad, best editor available under windows, after Crimson failing to start after I switched to Haskell mode. And don't even dare to mention Emacs, a lot of people don't like it.It's 2008 and creating a simple Haskell editor is not an rocket science. I need an integrated editor with syntax coloring and intellisense kind of reminding me how many and what type of arguments function takes. That's it. Another option is Visual Haskell, used as editor togather with winhugs, for interactivity. Also there is no debugger, so forgeth about stepping, functional style of Haskell, helps here but it's just a hassle debugging by hand.
Update :
Greg Buchholz adviced about GHCi debugger so I will check on it and advice later.Personally I would be happier with something for hugs.

6. Lacks reflexive abilities. Oh silly me of course it lack them. In Haskell everything is carved in stone. Programming in Haskell is like proving a mathematical theorem. There is no screwing around. It's like life in a monastery, full of fasting, preying and compassion under the all seeing eye of the abbot, punishing you for every mistake. So if this kind of life is for you, you'll end up in heaven, but if you want playing around and can't (don't) want life without women, parties, drinking and good food than Haskell is definately wrong choice.

7 and 8 are probably more personal than the others. But this whole post is personal. :)

7. Case sensitive - why does types has to start with uppercase and functions with lowercase. To make it easier for the implementation writers? Or to enhance readibilty. The latter has other way to be achieved. Any syntax coloring editor is smart enough to gave you visual clues what the hell some name means.

8. Indentation matters.

Saturday, January 5, 2008

Common lisp libraries - victims of drive by programming

In order to make an apple pie from scratch, you must first create the universe - Carl Sagan.

Recently I wanted to cook some embeded prolog, but modularly designed on top of some current lisp pattern matching library and add some tests.
So in the last few days I went shopping for regression test & pattern matching libraries. Among the illusion of abundance I encountered the known plague among the lisp(*) world, hackbraries (**).

Let me tell you first my criterias as different people have different expectations and mine are as fallows:
1. Work out of box
2. Portable across major implementations and OS(s)
3. Documentation with examples
4. Regression tests
5. asdf packaged

1st Whenever I choose using somebody's else code I expect to work out of the box.
If domain covered in the library was interesthing to me I would code it myself.
The reason to use somebody's else code is because I don't give damn about the implementation, I just want my problem solved.

I regularly work with Allegro & LW under Windows and Allegro & SBCL under linux, and I expect the library to work on all of them out of the box. If I had a Mac I would test with Mac and OpenMCL too. The correlation between portability and library quality is very strong.
Sticking to one implementation and OS is understandable if you're a vendor that want's to implement super performing library using implementation specific features, but if you're in the open source world keep in mind that number of lispers is relatively small, and after you divide them with major implementation / OS combinations the numbers would be far smaller.You might be the only one that could use it.
So stay away from implementation specific extensions as long as you can.

3. Documentation matters to help people understand what the library is actually doing and how is doing it. You are coding in different standard in a domain that I probably don't understand. Put some documentation so I could learn what do you achieved in your library , what's missing and how to use it.Also documentation offers another perspective why you wrote the code such that.

4. Tests are something that I run first when they exist, and they are good sign of library quality. Tests make sure that library works at least as author expected it to work. Also tests make a great learning material how to use the library. I learned more from hunchentoot and weblocks tests than from all documentation, examples and articles togather.

5. ASDF is de facto standard in the open source world, almost all the high quailty and popular libraries have it so fellow lispers expect it. Why bringing a disappointment for only a minute fo work.

So why does most of lisp libraries fail the above criterias while there is a relative sea of abundance?
I think it's partially because of nature of lisp and it's users. Lisp is a great language for rapid prototyping so if you need something, you could hack it yourself quickly, for an excellent example check this
link , it shows how
Tamas asked for a unit testing library and Frank Buss wrote one in 10 lines.
When all the functionality is only 10 lines, documentation,tests and asdf file doesn't make sense. As more functionality is needed, the code grows bigger untill our 10 liner expands to dozen of files and hundreds lines of code
thus becoming a *library*. If it's lucky it'll get an asdf file because that's easiest to add . The author and folks that use it from the beginning know it by hart so they don't need documentation nor tests. The *library* founds it's place in or sourceforge, and gets bugfixed for a while untill all the author(s)/*maintaineer* got hit by truck or lose interest. Lisp PR advertise it as a possibility, though most of them didn't even tried it. And everything is fine untill some outsider actually want to use this *library* for real. Soon enough he will found that it's easier for him to hack some 10-liner by himself instead of digging through somebody's else code. After more functionality is needed, the code grows bigger untill our 10 liner expands to dozen of files and hundreds lines of code thus becoming a *library*.
The vicious circle starts once again.

Lisp is like a pulley that will enable you to make code quicker than in any other language I tried, but you still you have to work hard to produce something than could be consumed by other. There are many high quality lisp libraries that are fine examples how things should be done the right way, but there are also ocean of halfbaked bundles of code that only waste your time.

If it ain't broke, maintain it


(*) When I say lisp I mean common lisp, for other uses I use lisp family of languages
(**) Hackbrary- Overgrown bundle of code lacking documentation, unit tests and maintenance, usually broken, posing as library.