The Rants, Raves, Gripes, and Prophecies of Paul R. Potts
Contents by Category
Contents by Date
I'm looking for people interested in meeting to talk about Lisp, Scheme, and other languages of that ilk. I have set up a meetup.com group, the Ann Arbor Lisp Languages Meetup; the group home page is here:
As of yet, we have not met, because no one has joined the goup. I'm posting this here in part so that anyone Googling for an Ann Arbor, Michigan, Lisp User's Group, or Scheme User's Group, will find it.
I would also like to get this topic syndicated via RSS onto the Planet Lisp aggregate site (planet.lisp.org), but as of yet the Lisp content is a little light, so I wait until there is a little bit more of interest here before asking them to syndicate my feed. The RSS feed for this topic is available as:
http://thepottshouse.org/blosxom.cgi/root/geeky/programming/lispish.rssThu, 17 Feb 2005 Destructuring
My friend Alan has been asking me questions about some Lisp programming idioms involving macros, such as destructuring-bind.
Destructuring, the idiom, is a technique in which the shape of a data structure can be determined at runtime, and the contents of the data structure bound to a set of variables.
More later.Wed, 16 Feb 2005 Programming Projects, Part 5
I've given up on VMWare, since I was not truly happy with it, and thanks to Paragon's Partition Manager, gotten my system set up in a decent configuration again, with enough disk space for both Linux and Windows 2000. So, for now, I'll just dual-boot.
My original G4 PowerBook, and my wife's iBook, both of year 2000 vintage, are both on their last legs, so we need to be ready to update our Mac infrastructure. We haven't quite figured out what to buy or, of course, how to pay for it.
I have not selected a project, but one that comes to mind now is editing iTunes file metadata. I've got all of Tolkien's works in audiobook form. The individual volumes such as the Fellowship of the Ring contain about 16 disks, and about 23 tracks each. Let's say I want to rename all of those 300+ tracks, as well as the disc (folder), following some pattern, and including the track x of y and disc m of n metadata in the file name, with the numbers padded so that the full filenames sort correctly. I'm doing it by hand now. Hmmm. I am not sure if iTunes exposes that much of itself to scripting, but if it does, it might be an interesting little trial problem.
Alan has been tossing me questions and ideas about Lisp. The latest was destructuring. I'll drop this "Programming Projects" topic for now and write some shorter notes on individual Lisp topics, although my ability to put time into hobby projects remains rather limited. Also, I really, really could use a brief vacation, preferably somewhere sunny, where my brain could absorb some photons and spend some time relaxing with my family. But alas, I don't have the vacation days. Maybe I can arrange something when this project is ended, although it may have to be unpaid time off. I'm almost willing to go that route.Mon, 31 Jan 2005 Programming Projects, Part 4
Hmmm. It has been a few days, and I have not had a lot of time at the computer. I did read part of Paul Graham's discussion of continuations, and ran some little demonstrations under PLT Scheme. More on that later.
I have not decided if I'm going to pay for VMWare, or whether I should give QEMU another shot. The Windows/Linux split is kind of a pain; I've wanted to leave the machine in Win2K instead of Linux so that Isaac can try out the contents of the Retro Gamer Magazine CD-ROMs. They've even provided a game construction set, and Isaac has built a little breakout game. I'm happy to see him doing something on the computer, even if it is mostly following a tutorial. So it stays in Win2K for now.
It would also be useful if I could really non-destructively resize and move all my partitions, both FAT, swap, and ext3. I should see if Knoppix has a tool to do that. I don't want to find out after the fact that it has trashed my drive beyond recovery, though. There isn't anything on the PC that is critical or original data and needs to be backed up, but just getting everything installed and configured is time-consuming. I'd hate to have to start over.
Maybe I should keep it in Win2K and set it up to run Linux under VMWare? Doable, but again, probably very time-consuming to set up. With a little more money I'd just put a second PC next to the first with a switch for the display, keyboard, and trackball. This could be useful if I ever pick up a Mac mini.
But anyway, I'm procrastinating. The projects are still on my radar. More later.Tue, 11 Jan 2005 Programming Projects, Part 3
This past weekend I attempted to make a little headway in some of the technical setup for the hobby and learning projects I've been discussing.
First, I brought my PC up to Fedora Core 3. This turned out to be much more painful than necessary; Firefox kept downloading an ISO for the first installation CD that was incomplete: it would silently fail, leaving an incomplete image file. The rest were fine. Strange. I didn't find out until booting it and doing the CD verification. The mirrors were running very slowly, and it took me multiple tries to get a clean ISO. So that was an enormous waste of time.
Despite all that, when I finally got the FC3 installer down, the upgrade proceeded fine. The exception was the patches. There is always an enormous backlog of patches, even for a recent distro; the patch process kept failing, with a variety of baffling error messages about missing RPMs that failed to tell me what went wrong. I finally had to apply the patches in batches (heh), rebooting after each batch; this worked. My 5 gig root and home partitions are strangely full; I need to clean out some build directories. I probably should have made my partitions larger. At some point I might start over on a second drive configured without dual-boot. Which brings me to...
One of the first steps was to make it so that I can keep my PC in Linux all the time, without booting back and forth to Windows 2000 just to run Word, Visio, or Functional Developer. After reading a rave review of the QEMU emulator in the British magazine Linux User, I decided to try it out. I was disappointed. It yields a segfault on startup consistently on FC3. Some other users are reporting the same problem on the forums. One guy with source tried to diagnose the problem with GDB. I don't really know how to use GDB off-hand. He had no luck working out a root cause and patch, so I figured that it was going to be a very time-consuming proposition, and not part of my core goals; I decided to try a different solution for now.
So, Bochs, another emulator mentioned in Linux User. This is ugly. It fails to have sensible default behavior, and uses an ugly configuration file. I got failures loading the VGA ROM image, for no reason I could ascertain. It would work with the demo Linux image, but the default config specifies a PC with 2 megabytes of RAM; that tells you how old it is. The user interface did not look promising, either, so I decided to move on.
I finally settled on downloading a 30-day trial of VMWare. For the most part, this just worked; it has quite slick virtual device drivers. I don't have a floppy drive, and it demands one each time I start the virtual machine. I'm not sure how to disable this; maybe in the virtual machine BIOS? I had a little bit of difficulty getting the VM to talk to my CD-RW drive, but a "legacy mode" checkbox makes it work.
It is not certified to work with Fedora Core 3, and Windows 2000 was unusable in full-screen mode due to drawing problems, but I loved the ability to set a wide variety of display sizes for the virtual video driver, and so could pick one that fit nicely on my 1600x1200 desktop. My biggest difficulties were in setting up a new partition in the uninitialized space on my hard drive to host the PC's disk image. I tried using the version of GNU parted that was on the Red Hat installation and rescue CD: it crashes, and fdisk was missing in action. If you run the installer, you can't get back to the Disk Druid partition editor without doing a complete reinstall. Trying will result in an unintentionally trashed GRUB boot (I use the NT bootloader to choose to load the Linux bootsector as a file; if it doesn't work, instead of the "LI" that happens with a failed LILO boot, you get a screen that says "GRUB" and nothing else. This failure seems to indicate that I need to replace the bootsector file with a new one extracted from the Linux boot volume. Fortunately, this fixed it.
I'm not sure of the name of the text-based, menu-driven partition editing tool I used before, but I don't think partedit was it. I got the latest stable source and built it successfully, although it gave me grief when my shell apparently could not find it. It finally started working, again for no apparent reason, and I was able to tell it that I wanted an ext3 partition and file system. No dice: it tells you that this feature doesn't work. (Then why is it one of the available options?) You have to specify ext2, and then use mkfs or something like that to write the file system, which can be ext3. Sigh. There's also a really ugly bit of user-interface design: you have to specify the partition point using a floating-point number of gigabytes. You specify a number, and it gets rounded, presumably, to the appropriate sector, which is displayed as a different floating-point number. You have to assume it is starting at the next available sector. Accidentally specifying a value that overlaps an existing partition produces a really ugly series of math errors, but then gets set to a legitimate value anyway. It's just bad. Sectors, bytes, etc., on a hard disk have precise integer values, not floating-point numbers of gigabytes; kilobytes, megabytes, and gigabytes are not really base-10 concepts anyway, so what does this even mean? I've no idea, but it is really the wrong way to display these values to the user.
Strangely, my Fedora Core 3 installation still imposes a 2 gigabyte limit on files. That isn't normally a problem, but building a virtual machine's hard disk image involves making a large file. It is not clear to me how to work around this; I tried setting some environment variables, but it didn't help. It is just baffling to me that this is still an issue; I guess that whatever hack-workaround-on-top-of-a-hack-workaround that fixes this limitation in the BIOS or IDE standard or whatever is still not part of the standard build. It seems that in 2005 it ought to be possible to set up a PC which, by default, doesn't have these strange arbitrary limitations (along with the cylinder restriction on the boot sector), but sadly, this hasn't yet happened.
It's also worth mentioning that the aphorism "Linux is free only if your time isn't worth anything" is still true, at least to some extent. I'm considering making the money-for-time tradeoff to buy VMWare, because it mostly Just Worked, where two open-source emulators didn't, and Wine doesn't.
Anyway. You can imagine the number of hours of my free time the above setup took. Without much free time left, I began to embark on attempting to get the jetty Java web server and sisc Scheme tools working.
I ran out of time, but the results were not encouraging. Ant would not compile either of them from source; apparently there is no JDK of any stripe standard in Fedora Core 3; there also seems to be no yum install option for said JDK. It comes with a GNU Java with a different name, and the standard Java is a script that acts as some kind of proxy mechanism. I've been away from Java for a few years, so I don't quite know what is going on with Java on Linux. I naively expected things to be more polished and plug-and-play than they were a few years ago.
There is fortunately a Sun JRE and JDK available as an RPM; I installed this successfully, but am now bogged down in config files. I apparently need to do something to configure the proxy scripts to use the Java that I want it to use, and wound up in a maze of twisty little man pages, all alike.
Anyway, there it stands. I'll see what I can do the next time I have a large chunk of free time. I've obviously been spoiled by the MacOS? X developer tools; everything pretty much just works. I'm also wondering if I made a mistake attempting to rely on Fedora, instead of another distribution like Debian. That's something guess I'll consider later; meanwhile, if I can get by with this configuration, I'll have to decide whether to pay for VMWare.
I'm getting Programmer's Paradise spam offering me special deals on VMWare. I'm quite confident I did not opt in to get junk e-mail when requesting the trial license key. This alone might just piss me off enough to refuse to do business with the VMWare folks -- I get so much junk mail already -- but if I go that route, I'll at least tell them why. Maybe QEMU will get a patch that makes it work with FC3.
While I'm at it, I should mention the source of some strange difficulty I was having building Gwydion Dylan. First, version 2.3.11 of d2c did not handle configuration on Fedora properly. This was because of a regular expression attempting to identify the CPU type, and failing to match. It's just another example of just how ad hoc all that autoconf stuff is. Debugging it is not easy, or at least if it is, I don't know how to do it. You've got to read code and insert print statements, and I'm not sure there's a better way. As a sort of 1.5 problem, apparently FC3 comes with an updated version of the autotools, which yields an endless series of warnings about deprecated quoting, so for some time I thought that this might be the problem. It generates warnings and errors about other programs, not just yours. Really, really ugly. I bought the book on the GNU autotools at one point and tried to get to know them; such a mess of m4, shell scripts, Perl, sed, and other mess does not seem worth becoming expert in unless absolutely necessary, like libtool.
The second problem was a weird error where apparently d2c could not find the GC library version 6. I don't know why it was attempting to load version 6; there isn't one. The real problem was apparently that I mis-installed a binary tarball starting at /usr/local instead of root (/). That was user error, but why this would result in the error I was getting is completely baffling; even if the buried binary d2c was being executed, or a buried library, none of them should have had a reference to a GC version 6. I only found the answer by accident, after days of full Mindy bootstraps and quite a bit of discussion on the #dylan IRC channel, installing a separate binary tarball for the GC library, etc. I try to be as careful as I can, but it is certainly easy to make a mistake like this, when your installation tool is as unhelpful as "tar."
I also made an attempt to see if I could make it easier to add content to my weblog. I tried to write a Ruby program that would talk to my IMAP mail server, read incoming messages sent to a specific account, get out the text, and deposit them in the weblog file tree.
Talking to the server and stepping through the "envelopes" (header information) was extremely simple. Extracting the actual text content seems impossible. There aren't any examples I could find. After a few hours spent reading a cascading series of Internet RFCs I concluded that the IMAP Ruby library is not exactly helpful with this; it seems like there ought to be some kind of basic, standard method to extract the plain text of the message. If there is, it isn't basic or simple, and the text part isn't visible in the data structures the library returns; there must be some kind of nested set of requests to get it from subsidiary return values, but I couldn't figure it out in a reasonable amount of time. So, so much for that quick hack. I'm not giving up entirely, which appears simpler, so maybe I'll try using POP. Or perhaps the right thing to do is to try to hook up a Ruby script to a procmail filter, although that seems like it could have security implications, although I'm not an expert in that area.
So, there it is: lots more to do; some progress on the infrastructure I want, but not a good start on any of the projects in particular. My approach is to work a bit on each one and see where I can get leverage, and repeat the next time I have some free time.Fri, 07 Jan 2005 Programming Projects, Part 2
OK, let me talk a little more about what I'd like to accomplish with my hobby programming projects. If you've read Zen and the Art of Motorcycle Maintenance, consider this a Chataqua (that term comes a little more naturally to me given that I grew up pretty close to Chataqua, New York, and have been to programs there a number of times).
First of all, the primary goal is to learn some new concepts and languages. It is awfully difficult to learn programming idioms and concepts in the abstract, so I'll need to set myself some small projects. In particular:
* Continuations. This is a big topic in the Scheme community, but in
most other languages continuations don't really exist. Nevertheless, the language research community considers continuations to be very important; they are considered an essential tool for compiling other languages, and now a number of people are talking about using continuations to simplify management of state in web applications. Nevertheless, they seem to be very difficult to explain: see recent threads on the Lambda the Ultimate weblog. I suspect that they are a little like closures and recursion in that they are easy to understand once you've used them for something, but experience with static languages actually hinders understanding. So, I'd like to use them for something. Perhaps a small web project using SISC. Perhaps for kicks I could then attempt to translate the same code into Common Lisp, which does not support continuations directly, using an implementation of continuations in Common Lisp, which might allow me to once and for all understand it, and also perhaps to port it again to Ruby, which supposedly features continuations as well.
* Macros. In the past I've attempted to write some Scheme macros, and
failed because of incompatibility of various Scheme implementations with the RSR5 standard. I've tried to write Dylan macros, but never quite got them right; the Dylan reference manual is not very useful as a tutorial, and it wasn't clear if the Gwydion Dylan implementation matched it properly. I suspect that I need to go back to Common Lisp and work my way through Paul Graham's On_Lisp, and then work from there to see if I can do the same thing in Dylan. I'd like to understand clearly all the varieties of macros and what people mean when they characterize a macro implementation as such-and-such, hygienic or not, functional or not, etc.
* Monads. Another concept that people on Lambda seem to have a great
deal of difficulty explaining. Haskell is probably the poster child of functional languages using monads.
* Scheme. I've used Scheme just enough to appreciate its simplicity
and usefulness; I've done some volunteer teaching, attempting to introduce grade school students to Scheme programming and recursion. It would be useful to dig a little deeper into Scheme to master continuations and macros in the language, and perhaps even write something useful in it.
* Common Lisp. I'm a little uncertain on this one; do I really want to
clutter my head with the huge extent of Common Lisp idioms and library functions? Should I consider Common Lisp to be a relic in and of itself, useful mainly for informing the design of newer languags, or the future, in which we'll just give up on less powerful languages and use Lisp? It seems a little difficult to keep Scheme and Lisp uploaded into my brain at the same time, kind of like switching between C and C++. But I can't deny the abstraction power of Common Lisp macros, and I'm really jonesing to work through Graham's book.
* Dylan. Yes, Dylan. Still my favorite multi-paradigm language.
Showing a few wrinkles and cracks, like a lack of native continuations, but still incredibly powerful if only for its multiple inheritance support, generic functions, multiple return values, full support for higher-order functions, and the possibilities of static typing and efficient compilation. (I keep hoping one day I can stop worrying about efficiency altogether, but that doesn't happen; what happens is that we try to take on bigger data sets and more complex algorithms). I keep hoping to find a piece of the Gwydion Dylan project where I can contribute something useful: even redoing error messages, or improving error checking, or basic documentation: these are the itches I'd like to scratch. But delving into the compiler is discouraging; I get frustrated by the compile times and daunting complexity of the Gwydion implementation's innards. The maintainers seem to have given up on Mindy, so there's an opportunity there: what would it take to write a tool that can boostrap a big Dylan implementation? (A hell of a lot, probably). I'm wondering if d2c really has a future. The Functional Objects compilers look a little more promising, but I haven't even gotten a toehold there, and I hate to have to keep rebooting to swap between Linux and Windows.
* Ruby. My favorite of the scripting languages, and the one that has
been the most enjoyable to use: it has a laughably simple syntax, takes ideas from many of my other favorite languages, and is refreshingly light on weird idioms, relying instead on a handful of common concepts like the code block (closure).
* Forth. Yes, Forth. If Lisp can be characterized as a "programmable
programming language," then Forth is another one. Aggresively simple in implementation, it is another one of the languages where the language gurus seem to expect you to be able to roll your own implementation. Forth is concise: both its great strength and great weakness, and enables metaprogramming, in its way. I've been doing a lot of embedded code work and looking at horribly complicated and error-prone implementations of relatively simple concepts. Much of this C code could be laughably simple in Forth, even the multi-threaded code; the code would probably shrink in size by at least a factor of ten, if not more. That's got to be worth exploring. I have not found a good implementation to try this out. I think it could run on a much less powerful embedded CPU than the one that it now requires. Even the exercise of refactoring into Forth words would surely teach me something about the refactoring I already attempt to practice.
* OCaml, Haskell, Erlang, D, Lua, etc. Perhaps worth looking at. So
many languages, so little time.
* NewtonScript?. To implement my own version; to play with. It could
be interesting to model the parent-proto inheritance concepts in a different language. If I could do something to contribute towards an open-source revival of the Newton, that would be great, but it is a little bit beyond my capabilities, what with the hardware engineering and prototyping.
If I could do some networking, produce some prototypes or code that I could use to get a new job, contribute to an open-source project, or use some of this code to further my weblog or other projects, that would be great too.
I'm looking at a few different languages now, primarily languages that allow at least some facility to create DSLs (Domain Specific Languages) and allow metasyntactic extension (macros). Here are the ideas, in no particular priority, broken down by language:
* Weblog Tools
I have a love-hate relationship with blosxom, my weblog software. It's Perl, which I loathe, but Perl made it very easy to install and allows it to use extensions in a uniform way. It's frustrating to extend and customize.
I've got a vague plan to build a weblog framework in Dylan, called Dybbler. I'd start by attempting to implement Markdown in Dylan. There's a basic web platform, called Koala. But I'm also told that it is not very far along, and that the startup time for d2c application code makes it frustrating to use compiled Dylan programs as CGI scripts.
It will also be difficult, if not impossible, to deploy a Dylan CGI on my ISP's site, at least without renting a dedicated server, so it would be an internal experiment for the time being. It might be possible to buy a fixed IP and host my own weblog at home, but I'm not sure I want to take that all on at once, what with security issues, spam and zombie attacks, and all that fun.
I'd like to arrange a better publishing mechanism. It would be nice if I could mail myself weblog entries and get them posted automatically. I spent a little time last night messing with my IMAP mail server and the Ruby IMAP support class. It was laughably easy to log in and walk through the messages and get header information, but the method to actually retrieve the message text seems to be either missing in action or obscure beyond belief.
In general, I'd say that the "let a thousand flowers bloom" world of web frameworks, Wikis, blogs, etc., is not really converging, something better is around the corner, and it isn't C# and it isn't Java.
* Embedded and Small Server Tools
I've got a bug up my rear end also about the need for more advanced tools in embedded systems. Java, including hardware Java implementations like ARM's Jazelle, seem promising, but as Peter Housel put it, once you've taken the red pill (using truly powerful languages), it is awfully hard to go back. A typical industrial embedded application is so mind-numbingly badly designed and over-implemented, it seems like there must be plenty of wasted memory and CPU to reclaim for the sake of using a rational toolset. One question I haven't seen answered adequately is whether I can build Dylan, Scheme, Common Lisp, or Ruby applications for a platform upon which I want to forbid not only garbage collection, but dynamic allocation altogether. Allocate everything on startup, and never cons, and never GC. It seems like this ought to be possible. Maybe a DSL Lisp dialect specifically for embedded code?
I'm also interested in seeing what I can get running on a Kuro Box ($160 micro-server platform). Could I get a JVM, SISC, web server, and database running on one of those things, to allow me to build a web application in Scheme, implementing state with continuations? What about using Common Lisp or Ruby on Rails? Any of these projects could turn into a useful article; all of them together could make a book.
* The Book
Then, there's the book idea. I've long wanted to write a book on Dylan programming and the Gwydion project. It might be more valuable to do a book on the Lisp family of dynamic languages for GUI, scripting, and web applications.
* And All the Rest
It seems like I'm proposing a lot. What I'm really trying to do is find my next job, one that I might enjoy doing. That's about it.
Oh, it would be nice to go back to school and get an advanced degree in computer science, but it would have to be a place like MIT or CMU that would teach me something useful about language implementation and design. And I'd have to get paid while I went, while still having time to raise my children. And I want to teach English, Math, Computer Science, and write science fiction too. And play Stick. Hmmm. One lifetime is starting to look like not quite enough. Maybe I could give up sleep. Maybe this is some kind of midlife crisis? I guess if you don't have goals you'll never get anything done; the key is deciding what to give up on.Tue, 04 Jan 2005 Programming Projects, Part 1
Now that our life is settling down slightly, I'm trying to see if I can set some learning goals in the area of learning, and possibly doing something useful with, Lisp-ish languages.
Last night I got SLIME + Emacs working with SBCL on my PC running Fedora Core 2. It is perhaps strange to say, but despite programming for so many years I've never learned Emacs, and in fact barely learned vi. I was reasonably proficient in vi at one point, when I used it to do some complicated multi-stage regexp-based search and replace, but whenever I'd try to dig into Emacs I'd quickly get hit with a wash of key remappings, startup scripts, and complications.
Key mappings are like obscure command-line switches that vary between UNIX dialects to me in that I tend to forget them rather quickly. I've finally used CVS often enough that I remember the basics, but most of my paid work has been with co-workers that don't tend to know, trust, or want to use command-line tools.
In my work developing drivers for MacOS? X I was using specific command line tools (like kextload and various kernel debugging tools) all the time, but it was really uphill convincing people to use CVS at all, even with a GUI front end, which had some annoying limitations. Trying to get the other developers to use CVS on the command line was a complete non-starter. Prior to that, in working on MacOS? 9 tools, there wasn't a command line (unless one wanted to use the somewhat orphaned MPW tool suite, which I didn't, preferring to use the Metrowerks CodeWarrior? suite that I was familiar with).
Apple also had, to some extent, a philosophy of leaving behind the command-line world with everything in text files, preferring binary data for file metadata and resources, and a scripting mechanism that tries to abstract out the particular scripting language itself and access application data via a hierarchy of callbacks done by unwinding something that could almost be considered binary s-expressions. I drank that Kool-Aid, writing code to support AppleScript?; it is all very interesting and very powerful, but also somewhat narrowly confined to its own niche, and like many ideas that are too complicated in execution for the average programmer to embrace, probably will remain so.
Believing that the whole world is Windows, though, many Windows programmers without wider experience are, if anything, even more provincial. They may not even be aware that their beloved Microsoft Visual Whatsis can be executed at the command line and that underneath it all there are basically Makefiles, albeit a species designed for non-portability.
It also probably has something to do with never having really teethed on UNIX when I was younger. Pre-college, I used word processors running on the TRS-80, Apple II, and Commodore 64; in college, I used (for coursework) VAX systems, running the VAX editor on VT100 or other terminals; for extra-curricular projects I used Mac systems, running what was variously called THINK or Lightspeed Pascal and C. During my senior year, I believe, the college put in a small UNIX box of some kind, but aside from becoming confused by key mappings and trying the various items in the games directory, I never really got the hang of it. Emacs always seemed like one of those infinitely configurable but endlessly complicated things like X-Windows; sure, I could somehow read mail and news and everything else in my editor, just like I could allegedly share all my directories between boxes and run GUI applications across the network, but there was rn and mail, so what was the point?
Anyway, I say this all by way of expressing my trepidation over embracing Emacs. I feel more comfortable in either the full-blown IDE mode with an environment like Macintosh Common Lisp, or directly at the command line. Emacs feels like something in between - excessively customizable, and excessively complicated. But if this is what is needed to learn Lisp macros, I'll give it a shot.
So what am I trying to accomplish again? More on that next time.