Project Mehitabel: Assembly Language, ELF, and the binutils

Last time we got a C toolchain working on RISC OS on a Raspberry Pi 3. That worked, but it also produced a Hello World that was 3 whole kilobytes. That’s awfully excessive for the kinds of programs we’ve written around here. We should be able to do better with assembly language and tighter integration with the OS’s executable formats.

ARM Assembly language written for RISC OS was traditionally written with a number of tools, but the clone of those tools in the GCC SDK is called asasm. Its syntax more or less matches the syntax described by ARM’s own reference material. Here’s a RISC OS Hello World in that format:

        AREA    |.text|,CODE,READONLY

        ENTRY
_start
        GLOBAL  _start
        SWI     &01                     ; OS_WriteS
        DCB     "Hello, world!",10,0
        ALIGN
        SWI     &11                     ; OS_Exit
        
        END

System calls in RISC OS turn out to be very much like the ones in MS-DOS or Linux. Replacing the x86’s INT instruction is called SWI (SoftWare Interrupt) or SVC (SuperVisor Call). Unlike MS-DOS and Linux, though, where one register traditionally identified which call was being made and the actual interrupt vector itself was fixed, RISC OS has a ton of different SWIs exposed, and they provide a huge amount of capability. These SWIs command not merely I/O and the filesystem, but the entire GUI (which originally lived in the ROM chip). It’s a very different world.

Anyway, Hello World is tremendously simple on this platform. There’s a SWI for printing out a zero-terminated string. That string, for this call, lives in the code, just after that call, and then once it hits the zero it stops printing, advances to the next 4-byte word boundary, and resumes your program execution at that point. We do that to print out Hello World and then follow it up with another SWI that ends the program.

Each instruction is 4 bytes long, so the code we’ve written here matches our previous record for Hello World on MS-DOS: this is 24 bytes, 8 of which are code.

We save this listing as a file named hello2 in a directory named s, much like how we handled our C source. We can then build, link, and run it with no special configuration (the linker expects a symbol named _start by default, and that’s what we provided):

*asasm hello2.s hello2.o
*ld -o hello2 hello2.o
*hello2
Hello, world!
*

Very simple. However, the resulting binary won’t work by default on non-Pi RISC OS machines. The reason is apparent in our output window:

The linker is actually producing its binary in the Unix-standard ELF format. RISC OS Pi include a kernel module for reading and interpreting ELF binaries, but that’s a later addition. The native executable format was called the Acorn Image Format, or AIF for short.

Furthermore, this ELF binary is 754 bytes, bigger than our Linux example, even.

We’ll address all this below the fold. Along the way, we’ll make use of some of the dustier corenrs of the UNIX utilities for working with binary files—the binutils.

Project Mehitabel: C Defying UNIX

Last time on Project Mehitabel, we wrote some programs in BBC BASIC and ran them out of the Filer or in the ShellCLI mode that lives outside of the desktop. This time, we’ll make some programs that run as machine code, and run them in the final type of text interface RISC OS provides: the task window. There’s nothing fancy here in terms of code; we’ll just be writing Hello World

What you’re really here for this time is to laugh at my suffering.

The design decisions RISC OS made long ago, and the ways that the community around it addressed those decisions to interoperate with the broader world, all have very ugly sharp corners. We’re going to make things work here, but this will also largely serve as a demonstration as to why what development there is for this platform is normally done with cross-compilers.

Step 1: Prep work

The first thing we need to do is run the !GCC app. (App bundles in RISC OS are directories with exclamation points as the first character in their names. There’s stuff that needs to be in the directory too, but the general mechanism is not unlike app bundles in the UNIX-based macOS versions.) We navigate to it through the Filer first by clicking the SD Card image in the icon bar and then navigating through to the appropriate directory:

As a collection of compilers and ancillary data, of course, gcc is not really an app; running it does nothing obvious and leaves no processes running. What this has done is extended the execution environment so that when we go into command-line mode it will be able to find the programs we need, and those programs will be able to find the data they need. RISC OS is notionally a co-operative multitasking system like Classic Mac OS or Windows 3.1, but there’s a very large amount of shared OS state across all applications. By setting environment variables, every program will see those changes immediately. That will include our command line.

But we aren’t ready to fire up the command line yet. As you may recall from last time, when we were playing with BASIC, we were handed a chunk of 640KB of RAM to work with. This was not a quirk of BASIC; it was the OS that allocated that memory to us. We won’t be able to run gcc in 640KB. To address this, we’ll left-click the little Raspberry Pi logo in the lower right. This calls up the Tasks Window:

This gives us a nice breakdown of memory usage in the system, including the two applications that are already running. (“Pinboard” is the application that manages the desktop icons, and “Snapper” is the screen-capture utility I’ve been using to illustrate these posts.) The green bars are fixed. The red ones are not. We can grab the “Next” bar with the mouse and drag it out to give the next program we start a bunch more memory. 64MB should do.

That done, we then middle-click the Raspberry Pi icon to open its context menu:

Most things in RISC OS can be middle-clicked to pop up a context menu, and you’ll need to do that in most apps to get anything done. As you’ve seen, it’s not like there are menu bars in these windows. As the menu shows, we could also hit Ctrl-F12 to open a new Task Window instead. That makes sense, because F12 alone is what suspends the desktop to put us into a CLI. Task Windows are text-only CLIs that live in a text editor window and are fully integrated with the CLI. If you aren’t working with non-GUI graphics modes, this is usually the way you’re going to interact with command-line applications.

Cynical and/or paranoid readers will notice that Shift Control F12 does nothing at all related to command lines and in fact powers down the system as immediately as possible. Don’t fat-finger that control-key, kids!

Anyway, with the Task Window created, we can dial back down the “Next” bar, and confirm that BASIC is getting allocated way more memory than it was before, and that we can run gcc inside the Task Window:

Everything that we have done to this point has just been so that we can get the compiler to run at all. A lot of Unix ports fit uneasily into RISC OS’s operational model, but this is enough to make you long for the simplicity of EDIT.COM and the Turbo C command-line build tools on DOS.

But that work is done! Let’s write some actual code. This part is also incredibly alien to anyone that’s used pretty much any other GUI-based OS, including that one X11 window manager written and configured in Haskell that doesn’t believe in the desktop metaphor.

Strap in, folks. This is the part that isn’t supposed to be wonky and weird.

Demo Release: Coast to Coast

I recently collaborated with the musician Nick Vivid on a Commodore 64 demo named “Coast to Coast” that was presented at @-Party 2017. It seems to be traditional to do writeups of one’s demos after the fact, but even if it weren’t, I’d be doing one. I’ve held off until the actual first presentation, though.

The Effects Used

The core techniques used here will start out looking pretty straightforward. There’s some text scrolling at the bottom of the screen, and the main part of the screen is consumed by text scrolling down at one pixel per frame. Then the 8 sprites are being used to cover the top of the screen with an additional marquee of scrolling text. (It take all 8 sprites to do this while keeping each letter static within the sprite; X-expanded sprites are 48 pixels wide, and the screen is 320 pixels across. So you need 368 pixels of space to scroll over the screen, but 7 only gives you 336.)

There’s nothing at all exciting about the visuals on this effect if you’ve been following this blog, but if you’ve been following this blog you also know that single-line splitscreen is a bit harder than it looks. Back in 2015 I didn’t really deal with the problem of scrolling the color RAM, either. I don’t address that here, either; I was just careful about my choices of colors.

Things get more fun about 20 seconds in, when I reveal that I wasn’t actually using the technique I just linked:

At this point I begin scrolling graphics up the screen as well as a down-scrolling background. Part of the reason I stuck to the simple sprite text scrolling was to show that I wasn’t doing this the easy way with sprites; this is a purely textual effect. The trick here is that while it looks like I’m scrolling one scanline down each frame, I’m actually using a much simpler technique to scroll up two lines every frame. The downscroll effect is created by combining that with rotating the character graphics down three lines each frame.

Of course, to get the objects that move up to actually move up, I also had to “coarse scroll” the screen up by one character row every four frames. This was handled via the traditional C64 technique of double-buffering the screen, spending the intervening four frames copying over all the characters, and then updating the video-matrix pointer as needed. I had a little jump table that would let me generate the map I was scrolling through a line at a time, and threading values through that jump table turned into a little miniature scripting language. So that was fun.

I did end up being a little more clever than I perhaps needed to be there. I was writing this while Nick Vivid was working on his music, and I didn’t know how expensive the music update would be each frame. I thus ended up restricting my coarse-scroll effects to the bottom line and then a selected (and variable) column of characters on the screen itself. In the end, I think this probably was overkill, but better to do the work and not need it than the other way around.

Speaking of the music, I had very little to do with that at all. He worked on his own with his musicmaking tools, produced a very nice piece, and I asked him if he could export it so that it would run only using RAM over location $3FFF. (The graphics and logic all live completely within the first 16KB, even uncompressed, even at runtime.) That was easily arranged and all I needed to do from there on out was copy it into place and then call into it at appropriate times.

With all the pieces together, all I did after that was run it through the PuCrunch system. This essentially turns a C64 binary into a self-decompressing, self-running archive. The home page has been gone for awhile, and while the code is still out there on the Internet, The Wayback Machine is the only way I can link the original site. PuCrunch was unreasonably effective at compressing this program; it dropped in size from about 12KB to less than 6.

Challenges, Compromises, and Goofs

Graphically, the biggest issue that I had was that I really, really did not want to have to wrangle color memory with things scrolling in three directions at once. I set multicolor text mode once, filled the entire screen with a single color, and then designed everything else around that. This meant that my scrolltext color also had to be in the range 8-15, and also had to be exactly the color 8 above whichever color I used for the text’s per-cell color. I wasn’t allowing myself any slack in the display, so that means (as we saw in the original vertical scrolling articles) that the scroll text’s color values would be shared with the last row of displayed graphical cells. Our change in scroll values would force a re-read, but the badline would come too quickly for us to do any reassignment. So I stuck with solid colors, and red/pink produced a reasonably nice display.

Conceptually the biggest problem that I had was that just scrolling stuff is kind of boring. Since one of the other goals of the demo was to be the invite for SynchroNY 2018, and that particular party takes place on a train that runs from NYC to Montreal, the train theme was set in pretty early. Discussions with Nick Vivid while we were talking about progress sort of developed the idea of it being about traveling in general, and once “remote entry to @-Party” was decided upon as the best venue to release in, that also brought in the notion of travel as a theme. The demo was being written, roughly, between SF and NYC, and about an event that would travel from NYC to Montreal. That’s when I got the idea of running the city names as extra text from, and it wasn’t too hard to hit enough major cities on one route to fill the time the music took up.

Unfortunately, those cities aren’t on an actual train route. If you want to take a train from the West Coast to NYC, you need to start in Los Angeles. I used Interstate 80’s route to select my interstitial cities.

An additional goof, which fortunately was caught before release, was that I ended up bungling the copying of the music code into place, and as a result a full third of it was originally missing. I think the three most devastating bugs that emerged during development and testing all ultimately were single-byte errors.

Reception

At the actual compo, the demo placed second, coming in behind an extremely elaborate demo for the Intellivision that is the most impressive work I’ve seen on the platform. Comments on it after the fact can be summarized as “not bad for a first effort”, which is about where it belongs. I’m not using any incredibly advanced techniques here and I’m not particularly talented as a graphician, either—the part of this project I like the most is that with only one extremely arguable exception, I’m getting an “impossible”-looking effect out of this without actually pushing the system’s limits. (The exception is that I do have to compensate for some of the VIC-II’s slightly inconsistent behavior surrounding vertical split-scrolling. I think that can count as “compensating for a quirk” rather than “pushing the envelope.”) This also let me put a lot of the small, independent routines I’d developed over the years into a single coherent program, and that was nice too.

Downloads

Here is the disk image that was sent to the competition.

Project Mehitabel: The (BBC) Basics

We’ve now got a RISC OS system up and running, and that also means that we’ve got our two most important applications ready: a web browser and Minecraft a text editor and Minesweeper:

Traditionally, one would program a RISC OS system in BBC BASIC, a dialect that originated on the BBC Micro and proceeded to evolve through the Archimedes line. It split off a number of times and has dialects that run on today’s more modern operating systems—the brandy system is the one I’ve found on most Linux machines—and while this isn’t a complete fool’s errand, BBC BASIC is primitive enough that I don’t think it would really be worth the effort to make new software in it for Linux or Windows.

That’s not to say that it’s not an interesting language, though. It’s capable of simply being used as a slight dialect of the early microcomputer BASICs that we’ve seen here before. It’s got its own quirks around file access and graphics, but most importantly, it’s case sensitive. Keywords and functions need to be in all capital letters, and because variables with capital letters often have important meanings to the system, accepted practice is to keep one’s own variables in lower case. We can see a copy of Hammurabi in my edit window up above.

We have several options for actually running the program. The one that is the most familiar to us would be to get into a command mode, start BASIC, and then LOAD and RUN our program. We can suspend RISC OS’s GUI and go into a command-line mode by pressing F12. This puts us at a command prompt not unlike a DOS prompt, from which we may start BASIC and then proceed as usual. Hilariously, as we do this, the GUI begins to scroll off the top of the screen:

Once we’re done, we enter QUIT at the BASIC prompt and return to the GUI by hitting ENTER at a blank line.

We don’t actually have to manually enter BASIC to do this, though. One of the advantages of the file metadata here is that the Hammurabi program has a file type of “BASIC program” (&FFB—RISC OS uses the ampersand to indicate hexadecimal constants) and as such we can simply type Hammurabi at the ShellCLI prompt and be in our program.

We can also run it without leaving the GUI by double-clicking the program in the Filer (what RISC OS calls the Explorer or Finder). Hammurabi isn’t a GUI program, so this still takes over the desktop, but it does it in a somewhat more friendly way:

That window looks like it should be movable around the desktop, but it actually isn’t. The mouse doesn’t do anything useful while you’re running a program in this way. If you really want to properly integrate with the GUI, and you’re a purely-textual application, the right thing to do is to run it in a “task window.” We’ll get to those in a later post, once we start working with C.

But BBC BASIC is actually much more sophisticated than running a reformatted version of the C64 BASIC Hammurabi suggests.

Kicking off Project Mehitabel

Sorry for the radio silence, folks. I’ve both had less time to work on hobby projects of late, and the projects I was working on weren’t the kind that I could write up as they went. Also, I’ve pushed my knowledge of how the systems of my youth worked to the point that it feels like all else is commentary.

That ends up being pretty demotivating for a blog like this. Except for the DOS work, all the projects I’ve written about on this blog have involved 8-bit systems. One of the main reasons for that is that they feel like “lost technology.” The capabilities each has are unique and they are all programmed very differently, even when they use the same chip. DOS lives a bit in that space too, slowly growing out of it as you move into the Windows 9x era. On the other hand, once you start looking at the 32-bit systems—and remember, the 32-bit era begins in 1984 with the Macintosh, followed up over the next three years by machines like the Amiga, the Atari ST, and the Acorn Archimedes—it feels less like you are dealing with unique relics the likes of which the modern world has never seen, and more like dealing with modern systems that are heavily stripped down. That’s a fundamentally less interesting question to me, and boils down to constrained use of the same dev tools we’ve been using in the UNIX world for 40 years or more.

If you’re interested in early GUI programming, these are a gold mine, of course, and the skills you’ve picked up as a systems programmer in the modern world suddenly end up applicable to surprising machines. I can see that being fun too, but without any history with any of these systems, there isn’t much of the same kick to it.

But there’s one fun little wrinkle on this. You know what else is basically a modern system that’s been heavily stripped down?

A Raspberry Pi, that’s what. You can make a very strong case that the Pi is the true inheritor of the Acorn Archimedes and as such the BBC Micro:

• It’s very British.
• It was originally intended for primarily educational purposes.
• In particular, it was intended to be programmed by its end user.
• It’s based on the same chip series as the Archimedes and its successors.
• RISC OS, the operating system and GUI that was released alongside the Archimedes and which continued through its successors, was ported to the Raspberry Pi within a year of its launch.

I recently got a Pi 3. Building a RISC OS system and writing some software for it—or porting some software to it—sounds like a fun thing to do. I’ve named this “Project Mehitabel” after the name of Archie’s old companion, and because getting this all to work has been a bit like herding cats.

How to Not Learn Assembly Language in DOS, Part 2

Last time, we talked about writing memory directly and invoking interrupts. This time, we’ll cover writing I/O ports directly and writing our own interrupts.

The Project

Our first draft of the Smoking Clover program relied on the delay() routine in Borland’s DOS extensions to control animation speed, and used BIOS interrupts to rewrite the palette. This produces very inconsistent animation depending on CPU speed. In this article we will reorganize the animation to be governed by IRQ0, the timing interrupt. This also means that, since we’re changing the palettes inside an interrupt, we can’t use interrupts to do it—we’ll need to use the VGA chip’s I/O ports to alter colors. We’ll also need to actually properly program the timer. In the end, we should also have a reusable timer callback system.

Code begins below the break.

How to Not Learn Assembly Language in DOS, Part 1

After a couple of weeks of ZX81 work, let’s jump ten years forward in time and port Bill Gosper’s Smoking Clover effect to DOS and VGA. Basically every description I’ve seen of this, including the one on Wiki, is some variation of the minimal definition in ESR’s old Jargon File:

Many convergent lines are drawn on a color monitor in such a way that every pixel struck has its color incremented…. The color map is then repeatedly rotated.

The latest edition of the Jargon File actually specifies that the lines are drawn such that one endpoint is in the middle of the screen and the others are one pixel apart around the perimeter of a large square. This is more or less what I attempted when I first implemented the effect back in high school—but doing this turns out not to produce the clover effect. You get a more tunnel-like effect instead.

What worked for me was to have my endpoints march, exactly pixel at a time, around a circle much larger than the display screen. This in turn isn’t exactly the same thing as points equally spaced along the perimeter of a circle, either—what I ultimately did was solve x² + y² = r² for successive values of x, in exactly the range where x <= y. I then reflected that line across the X and Y axes as well as the line y=x. This finally produced the result I wanted:

Somewhat unusually for this blog, this program is actually written with no use of assembly language whatsoever. Back when I was exploring the HAT function, I demonstrated how to invoke interrupts and memory-mapped I/O in Turbo Pascal 5.5.

In this article I will bring C up to speed. Like my earlier work here, this will be written using Borland’s Turbo C 2.0.1, still available free of charge from Embarcadero Software.

What We Need

So, aside from a perverse breed of bragging rights, what does assembly language buy us, anyway? Mostly, what it buys us is direct access to the hardware:

• We can trap to the operating system, firmware, or hypervisor. In DOS we do that with the INT instruction. We’ve used this all over our DOS work here, for everything from changing graphics modes to printing messages to exiting the program.
• We may directly read and write video memory or other kinds of memory-mapped I/O. In DOS, we’ve used this just to control graphics memory—on the 6502-based systems, it’s how we controlled all our peripherals.
• We may directly read and write the I/O ports. Not all chips have these—the 6502 and its cousins in particular do not— but they figure heavily in both the Z80 and the x86 line of chips. We didn’t touch the I/O ports on the ZX81, but we’ve used them in DOS to produce nonstandard CGA modes and to program the Adlib and Sound Blaster chips.
• We may override the operating system’s own interrupt vectors and provide our own. We used this when we coerced digital sound out of the PC speaker.

All of these things are possible within Borland’s HLLs. In this article, we’ll focus on the first two, and leave the last two for another time.