The Rants, Raves, Gripes, and Prophecies of Paul R. Potts

Contents by Category

Contents by Date

Favorite Links

My Wiki: main entrance
Boing Boing
Gwydion Dylan
Paul Graham
Richard P. Gabriel

Fri, 26 Aug 2005 Installer VISE

Installer VISE has a good reputation as one of the best packages for building installers, but apparently that is not saying much... it is really pretty awkward to use, and the documentation is weak.

It is like writing a program in a conventional language where all the actual behavior is hidden. Like: what the conditionals actually say; how comparisons are done (what constitutes true and false); not being able to evaluate any kind of previous step for success or failure without introducing a superfluous variable test item; not being able to actually manage types and perform type coercion yourself; not being able to actually manage strings yourself... ugh.

It does supply quite a bit of behavior for you, though, which would be quite awkward if you had to write it all yourself. Oh well...

[/root/geeky/programming] permanent link

Wed, 06 Jul 2005 Handling SREC files with Ruby

Ruby is definitely one of my favorite languages, and I am far more comfortable with it than I am with Perl. It seems very expressive; very frequently, once I find the right library method, the code pretty much writes itself.

However, just like with Perl, there is more than one way to do it, so I often find myself looking for more appropriate idioms for the task at hand. There is also some question of efficiency; I'm not trying to optimize prematurely, but the program I'm working on runs slower than I'd like, and I think Ruby can do better.

To start, I'll just share one quick annoyance. The first is that there is some deficiency in handling of typing such that statements like

puts "Checksum: " + chksm

will not work; you get a run-time error (can't convert Fixnum into String)." This seems wrong; it is not very "DWIM" (do what I mean) when considered in light of Ruby's philosophy of weak "duck typing" -- if it quacks, for all practical purposes your program can treat it as a duck, or in this case a string. I bring this up because I keep getting this error -- I have a habit of forgetting to add .to_s to the variable. I'm thinking in terms of C++ iostreams, where the type is taken care of using the stream << operator.

But on to a meatier question. I want to be able to treat a string containing ASCII hex digits as an array of bytes or as a Fixnum (where I can specify the byte ordering). For example, I want to be able to turn

"DEADBEEF" into an array of unsigned integers [0xDE, 0xAD, 0xBE, 0xEF] (for purposes of generating a bytwise checksum), or into the unsigned integers 0xDEAD, 0xDEADBE, or 0xDEADBEEF (depending on the record type I'm working on).

These are the ASCII character values (exprssed in hex); 'D' is decimal 68, hex 44. Not very useful; I get 2 strings of ASCII hex out. These aren't very useful for translation, since I'd have to turn them into integers, translate them to numeric values (not just a simple offset, because the hex characters are not contiguous) and then assemble the high and low nybbles into byte values using (high << 4) + low, or some such.

There must be a better way to do this. Here's my first try:

def make_checksum (str)
  checksum = 0
  0.step(str.length - 1, 2) { |idx|
    checksum += (str[idx].chr.hex * 16) + (str[idx + 1].chr.hex)
  return (~checksum) & 0xFF

Ugh. Look at the way I have to access the characters: str[idx] returns numeric types, not character types, which don't have a hex() method, so I have to convert them from the original characters in the string to integers, to characters, and then apply hex() to that.

Another way is to make substrings, I suppose, but it doesn't perform better, probably because it is generating a lot of extra string objects:

def make_srec_checksum (str)
  puts "make_srec_checksum; str is: " + str
  checksum = 0
  0.step(str.length - 1, 2) { |idx|
    checksum += str[idx..(idx + 1)].hex
  return (~checksum) & 0xFF

Another is that the String.unpack() method seemed from reading the documentation that it had exactly what I needed; it appeared that the "h" and "H" characters in the control string for the unpack method would be capable of unpacking ASCII hex data like DEADBEEF into byte values. That seems to be what unpack is all about. Instead, it generates ASCII hex bytes.

$ ruby -e 'puts "0123456789ABCDEF".unpack("H2" * 16)'


Not what I wanted. Most likely, there is a much better way!

It might be interesting to compare Common Lisp and Ruby implementations of my SREC tool; how concise, yet expressive, can I be in each language? This also might serve to shed some light on whether Common Lisp has fallen behind as far as libraries for typical file-handling and scripting-type tasks. More on that later.

[/root/geeky/programming/ruby] permanent link

Mon, 21 Feb 2005 Lisp Group

I'm looking for people interested in meeting to talk about Lisp, Scheme, and other languages of that ilk. I have set up a group, the Ann Arbor Lisp Languages Meetup; the group home page is here:

As of yet, we have not met, because no one has joined the goup. I'm posting this here in part so that anyone Googling for an Ann Arbor, Michigan, Lisp User's Group, or Scheme User's Group, will find it.

I would also like to get this topic syndicated via RSS onto the Planet Lisp aggregate site (, but as of yet the Lisp content is a little light, so I wait until there is a little bit more of interest here before asking them to syndicate my feed. The RSS feed for this topic is available as:

[/root/geeky/programming/lispish] permanent link

Thu, 17 Feb 2005 Destructuring

My friend Alan has been asking me questions about some Lisp programming idioms involving macros, such as destructuring-bind.

Destructuring, the idiom, is a technique in which the shape of a data structure can be determined at runtime, and the contents of the data structure bound to a set of variables.

More later.

[/root/geeky/programming/lispish] permanent link

Wed, 16 Feb 2005 Programming Projects, Part 5

I've given up on VMWare, since I was not truly happy with it, and thanks to Paragon's Partition Manager, gotten my system set up in a decent configuration again, with enough disk space for both Linux and Windows 2000. So, for now, I'll just dual-boot.

My original G4 PowerBook, and my wife's iBook, both of year 2000 vintage, are both on their last legs, so we need to be ready to update our Mac infrastructure. We haven't quite figured out what to buy or, of course, how to pay for it.

I have not selected a project, but one that comes to mind now is editing iTunes file metadata. I've got all of Tolkien's works in audiobook form. The individual volumes such as the Fellowship of the Ring contain about 16 disks, and about 23 tracks each. Let's say I want to rename all of those 300+ tracks, as well as the disc (folder), following some pattern, and including the track x of y and disc m of n metadata in the file name, with the numbers padded so that the full filenames sort correctly. I'm doing it by hand now. Hmmm. I am not sure if iTunes exposes that much of itself to scripting, but if it does, it might be an interesting little trial problem.

Alan has been tossing me questions and ideas about Lisp. The latest was destructuring. I'll drop this "Programming Projects" topic for now and write some shorter notes on individual Lisp topics, although my ability to put time into hobby projects remains rather limited. Also, I really, really could use a brief vacation, preferably somewhere sunny, where my brain could absorb some photons and spend some time relaxing with my family. But alas, I don't have the vacation days. Maybe I can arrange something when this project is ended, although it may have to be unpaid time off. I'm almost willing to go that route.

[/root/geeky/programming/lispish] permanent link

Mon, 31 Jan 2005 Programming Projects, Part 4

Hmmm. It has been a few days, and I have not had a lot of time at the computer. I did read part of Paul Graham's discussion of continuations, and ran some little demonstrations under PLT Scheme. More on that later.

I have not decided if I'm going to pay for VMWare, or whether I should give QEMU another shot. The Windows/Linux split is kind of a pain; I've wanted to leave the machine in Win2K instead of Linux so that Isaac can try out the contents of the Retro Gamer Magazine CD-ROMs. They've even provided a game construction set, and Isaac has built a little breakout game. I'm happy to see him doing something on the computer, even if it is mostly following a tutorial. So it stays in Win2K for now.

It would also be useful if I could really non-destructively resize and move all my partitions, both FAT, swap, and ext3. I should see if Knoppix has a tool to do that. I don't want to find out after the fact that it has trashed my drive beyond recovery, though. There isn't anything on the PC that is critical or original data and needs to be backed up, but just getting everything installed and configured is time-consuming. I'd hate to have to start over.

Maybe I should keep it in Win2K and set it up to run Linux under VMWare? Doable, but again, probably very time-consuming to set up. With a little more money I'd just put a second PC next to the first with a switch for the display, keyboard, and trackball. This could be useful if I ever pick up a Mac mini.

But anyway, I'm procrastinating. The projects are still on my radar. More later.

[/root/geeky/programming/lispish] permanent link

Tue, 11 Jan 2005 Programming Projects, Part 3

This past weekend I attempted to make a little headway in some of the technical setup for the hobby and learning projects I've been discussing.

First, I brought my PC up to Fedora Core 3. This turned out to be much more painful than necessary; Firefox kept downloading an ISO for the first installation CD that was incomplete: it would silently fail, leaving an incomplete image file. The rest were fine. Strange. I didn't find out until booting it and doing the CD verification. The mirrors were running very slowly, and it took me multiple tries to get a clean ISO. So that was an enormous waste of time.

Despite all that, when I finally got the FC3 installer down, the upgrade proceeded fine. The exception was the patches. There is always an enormous backlog of patches, even for a recent distro; the patch process kept failing, with a variety of baffling error messages about missing RPMs that failed to tell me what went wrong. I finally had to apply the patches in batches (heh), rebooting after each batch; this worked. My 5 gig root and home partitions are strangely full; I need to clean out some build directories. I probably should have made my partitions larger. At some point I might start over on a second drive configured without dual-boot. Which brings me to...

One of the first steps was to make it so that I can keep my PC in Linux all the time, without booting back and forth to Windows 2000 just to run Word, Visio, or Functional Developer. After reading a rave review of the QEMU emulator in the British magazine Linux User, I decided to try it out. I was disappointed. It yields a segfault on startup consistently on FC3. Some other users are reporting the same problem on the forums. One guy with source tried to diagnose the problem with GDB. I don't really know how to use GDB off-hand. He had no luck working out a root cause and patch, so I figured that it was going to be a very time-consuming proposition, and not part of my core goals; I decided to try a different solution for now.

So, Bochs, another emulator mentioned in Linux User. This is ugly. It fails to have sensible default behavior, and uses an ugly configuration file. I got failures loading the VGA ROM image, for no reason I could ascertain. It would work with the demo Linux image, but the default config specifies a PC with 2 megabytes of RAM; that tells you how old it is. The user interface did not look promising, either, so I decided to move on.

I finally settled on downloading a 30-day trial of VMWare. For the most part, this just worked; it has quite slick virtual device drivers. I don't have a floppy drive, and it demands one each time I start the virtual machine. I'm not sure how to disable this; maybe in the virtual machine BIOS? I had a little bit of difficulty getting the VM to talk to my CD-RW drive, but a "legacy mode" checkbox makes it work.

It is not certified to work with Fedora Core 3, and Windows 2000 was unusable in full-screen mode due to drawing problems, but I loved the ability to set a wide variety of display sizes for the virtual video driver, and so could pick one that fit nicely on my 1600x1200 desktop. My biggest difficulties were in setting up a new partition in the uninitialized space on my hard drive to host the PC's disk image. I tried using the version of GNU parted that was on the Red Hat installation and rescue CD: it crashes, and fdisk was missing in action. If you run the installer, you can't get back to the Disk Druid partition editor without doing a complete reinstall. Trying will result in an unintentionally trashed GRUB boot (I use the NT bootloader to choose to load the Linux bootsector as a file; if it doesn't work, instead of the "LI" that happens with a failed LILO boot, you get a screen that says "GRUB" and nothing else. This failure seems to indicate that I need to replace the bootsector file with a new one extracted from the Linux boot volume. Fortunately, this fixed it.

I'm not sure of the name of the text-based, menu-driven partition editing tool I used before, but I don't think partedit was it. I got the latest stable source and built it successfully, although it gave me grief when my shell apparently could not find it. It finally started working, again for no apparent reason, and I was able to tell it that I wanted an ext3 partition and file system. No dice: it tells you that this feature doesn't work. (Then why is it one of the available options?) You have to specify ext2, and then use mkfs or something like that to write the file system, which can be ext3. Sigh. There's also a really ugly bit of user-interface design: you have to specify the partition point using a floating-point number of gigabytes. You specify a number, and it gets rounded, presumably, to the appropriate sector, which is displayed as a different floating-point number. You have to assume it is starting at the next available sector. Accidentally specifying a value that overlaps an existing partition produces a really ugly series of math errors, but then gets set to a legitimate value anyway. It's just bad. Sectors, bytes, etc., on a hard disk have precise integer values, not floating-point numbers of gigabytes; kilobytes, megabytes, and gigabytes are not really base-10 concepts anyway, so what does this even mean? I've no idea, but it is really the wrong way to display these values to the user.

Strangely, my Fedora Core 3 installation still imposes a 2 gigabyte limit on files. That isn't normally a problem, but building a virtual machine's hard disk image involves making a large file. It is not clear to me how to work around this; I tried setting some environment variables, but it didn't help. It is just baffling to me that this is still an issue; I guess that whatever hack-workaround-on-top-of-a-hack-workaround that fixes this limitation in the BIOS or IDE standard or whatever is still not part of the standard build. It seems that in 2005 it ought to be possible to set up a PC which, by default, doesn't have these strange arbitrary limitations (along with the cylinder restriction on the boot sector), but sadly, this hasn't yet happened.

It's also worth mentioning that the aphorism "Linux is free only if your time isn't worth anything" is still true, at least to some extent. I'm considering making the money-for-time tradeoff to buy VMWare, because it mostly Just Worked, where two open-source emulators didn't, and Wine doesn't.

Anyway. You can imagine the number of hours of my free time the above setup took. Without much free time left, I began to embark on attempting to get the jetty Java web server and sisc Scheme tools working.

I ran out of time, but the results were not encouraging. Ant would not compile either of them from source; apparently there is no JDK of any stripe standard in Fedora Core 3; there also seems to be no yum install option for said JDK. It comes with a GNU Java with a different name, and the standard Java is a script that acts as some kind of proxy mechanism. I've been away from Java for a few years, so I don't quite know what is going on with Java on Linux. I naively expected things to be more polished and plug-and-play than they were a few years ago.

There is fortunately a Sun JRE and JDK available as an RPM; I installed this successfully, but am now bogged down in config files. I apparently need to do something to configure the proxy scripts to use the Java that I want it to use, and wound up in a maze of twisty little man pages, all alike.

Anyway, there it stands. I'll see what I can do the next time I have a large chunk of free time. I've obviously been spoiled by the MacOS? X developer tools; everything pretty much just works. I'm also wondering if I made a mistake attempting to rely on Fedora, instead of another distribution like Debian. That's something guess I'll consider later; meanwhile, if I can get by with this configuration, I'll have to decide whether to pay for VMWare.

I'm getting Programmer's Paradise spam offering me special deals on VMWare. I'm quite confident I did not opt in to get junk e-mail when requesting the trial license key. This alone might just piss me off enough to refuse to do business with the VMWare folks -- I get so much junk mail already -- but if I go that route, I'll at least tell them why. Maybe QEMU will get a patch that makes it work with FC3.

While I'm at it, I should mention the source of some strange difficulty I was having building Gwydion Dylan. First, version 2.3.11 of d2c did not handle configuration on Fedora properly. This was because of a regular expression attempting to identify the CPU type, and failing to match. It's just another example of just how ad hoc all that autoconf stuff is. Debugging it is not easy, or at least if it is, I don't know how to do it. You've got to read code and insert print statements, and I'm not sure there's a better way. As a sort of 1.5 problem, apparently FC3 comes with an updated version of the autotools, which yields an endless series of warnings about deprecated quoting, so for some time I thought that this might be the problem. It generates warnings and errors about other programs, not just yours. Really, really ugly. I bought the book on the GNU autotools at one point and tried to get to know them; such a mess of m4, shell scripts, Perl, sed, and other mess does not seem worth becoming expert in unless absolutely necessary, like libtool.

The second problem was a weird error where apparently d2c could not find the GC library version 6. I don't know why it was attempting to load version 6; there isn't one. The real problem was apparently that I mis-installed a binary tarball starting at /usr/local instead of root (/). That was user error, but why this would result in the error I was getting is completely baffling; even if the buried binary d2c was being executed, or a buried library, none of them should have had a reference to a GC version 6. I only found the answer by accident, after days of full Mindy bootstraps and quite a bit of discussion on the #dylan IRC channel, installing a separate binary tarball for the GC library, etc. I try to be as careful as I can, but it is certainly easy to make a mistake like this, when your installation tool is as unhelpful as "tar."

I also made an attempt to see if I could make it easier to add content to my weblog. I tried to write a Ruby program that would talk to my IMAP mail server, read incoming messages sent to a specific account, get out the text, and deposit them in the weblog file tree.

Talking to the server and stepping through the "envelopes" (header information) was extremely simple. Extracting the actual text content seems impossible. There aren't any examples I could find. After a few hours spent reading a cascading series of Internet RFCs I concluded that the IMAP Ruby library is not exactly helpful with this; it seems like there ought to be some kind of basic, standard method to extract the plain text of the message. If there is, it isn't basic or simple, and the text part isn't visible in the data structures the library returns; there must be some kind of nested set of requests to get it from subsidiary return values, but I couldn't figure it out in a reasonable amount of time. So, so much for that quick hack. I'm not giving up entirely, which appears simpler, so maybe I'll try using POP. Or perhaps the right thing to do is to try to hook up a Ruby script to a procmail filter, although that seems like it could have security implications, although I'm not an expert in that area.

So, there it is: lots more to do; some progress on the infrastructure I want, but not a good start on any of the projects in particular. My approach is to work a bit on each one and see where I can get leverage, and repeat the next time I have some free time.

[/root/geeky/programming/lispish] permanent link

You are here: root :: geeky :: programming You are viewing entries 1 to 7 of 13. next 6

Creative Commons License

Viewable With Any Browser

You are here:
root :: geeky :: programming

You are viewing entries 1 to 7 of 13.

next 6