April 30, 2017

Development of Aboriginal Linux has ended, replaced by mkroot.

Aboriginal Linux hasn't had a release for a year because of toolchain issues, which gradually led to a design rethink, resulting in a simpler design which is a better base for making Android self-hosting.

The original motivation of Aboriginal Linux was that back around 2002 Knoppix was the only Linux distro paying attention to the desktop, and Knoppix's 700 megabyte live CD included the same 100 megabytes of packages Linux From Scratch built, which provided a roughly equivalent command line experience as the 1.7 megabytes tomsrtbt which was based on busybox and uClibc. If I could free up an extra 100 megs of space on Knoppix's boot CD they'd work wonders with it, so I started work trying to build a fully functional Linux From Scratch with all the gnu bloatware replaced with busybox and uClibc.

The quest for the smallest Linux system capable of rebuilding itself under itself (and building the original Linux From Scratch under the result as a proof of equivalent functionality) yielded a system built from 7 packages: linux, busybox, uClibc, gcc, binutils, make, and bash. With a build control image that build Linux From Scratch, the project had met its initial design goals and had its 1.0 release way back in 2010.

Along the way I did so much work on BusyBox I wound up maintaining it, and cleaning up after Busybox' "hall of shame" made me care more about licensing issues than most people, which is why when GPlv3 happened I said no. And I similarly refused to ever ship a GPLv3 package in Aboriginal Linux, which meant I had to maintain the last GPLv2 release of gcc and binutils, patching my way around breakage every time a new kernel came out that depended on a new feature.

I managed this for about a decade, but after Linux 4.3 the next two kernel releases broke four different architectures, which was more than I had spare time to deal with just then. Strategically, switching to clang/llvm/lld would be a better investment than more work on the old gplv2 toolchain, but it was missing support for most architectures (including superh, which I work on elsewhere. Once upon a time I'd hoped to move to qcc but making Android self-hosting became a more important use of time. This left Aboriginal Linux with no way forward that wasn't either a lot of work to produce any result, or a waste of time in the long run.

So the project lay dormant until the musl-libc maintainer created his own toolchain builder, and a new (simpler) design became possible. With a little persuasion, he built not only cross compilers but portable _native_ compilers for the supported targets, which was the thing I'd never managed to get out of any external project before. (The closest was Debian, which dynamically links its compilers against the host's glibc, and doesn't support all targets.)

The new design provides a better test harness for toybox, and better serves the goal of making Android self-hosting. The Android guys are working on their own toolchains based on LLVM, and Rich has plans to host binaries once I've tested all the targets and shown that what we're building is good enough. (I refuse to distribute them, but others are free to take that risk.)

Given a choice between shipping GPLv3 binaries and ending Aboriginal Linux, it was no contest. And thus the project ended.

March 30, 2016

The repository has moved to github, both for this project and for the associated build control images.

January 11, 2016

Aboriginal Linux 1.4.5 is out, based on hg 1836, using Linux 4.3.

It's just a kernel version upgrade, no other changes. We've been consistently running a couple kernel versions behind, and I'd like to catch up.

January 7, 2016

Aboriginal Linux 1.4.4 is out, based on hg 1833, using Linux 4.2. This release patches gcc 4.2.1 to build under gcc 5.2, fixes the sh2eb build, adds proper powerpc-440fp support (running under QEMU's -M bamboo board), and converts most architectures (armv4tl, armv5l, armv6l, i486, i586, i686, mips, mipsel, powerpc, sh4, x86_64) to use musl instead of uClinux.

Dreamhost deleted a big chunk of history out of the mailing list web archive again. Last year it was just a few weeks, this time they retroactively zapped 11 months (almost all of 2015). I provided some details in another affected project, but given that last time they never filled in the previous gap in the archive, I don't hold out much hope. If history is a guide, they'll keep apologizing as long as I keep poking them about it, but nothing will ever get fixed. (Some of the vanished posts are in archive.org.)

Due to the musl conversion, new build control images are necessary to provide dropbear, strace, and linux from scratch. (Both strace-4.9 and Linux From Scratch 6.8 fail to build under musl for basically stupid reasons, in the case of LFS more than one package actually has an #if/else staircase for every known libc type ending in an #error if it doesn't recognize it. Since musl doesn't like and pretend to be glibc like uClibc does, I need to upgrade package versions...) Those aren't uploaded yet, but should be available soon.

The system-image.sh build is now called every time by build.sh and handles its own dependencies: it rebuilds the rootfs, native compiler, and kernel images as necessary. (This means if you tweak the root filesystem contents, system-image.sh doesn't have to rebuild the kernel each time unless you're using SYSIMAGE_TYPE=rootfs instead of the default SYSIMAGE_TYPE=cpio so faster build and test cycles.)

Several new config variables:

• MY_PATCH_DIR - lets you specify an additional patch directory applied after sources/patches (in this case with one subdirectory per package instead of the package name in the prefix, so sources/patches/linux-thingy.patch would be patchdir/linux/thingy.patch instead).

• MY_ROOT_OVERLAY - replaces SIMPLE_ROOT_OVERLAY, still a directory of files to cp -a into root-filesystem.

• MY_CROSS_PATH and MY_CROSS_PREFIX - specify an external cross compiler. Providing a native compiler in that case (via MY_ROOT_OVERLAY) is your problem, native-compiler.sh generally won't work without a corresponding cross compiler built from the same source.

You can now specify EXTRACT_ALL=force to populate the page cache (extract tarball and apply patches) even when the packages directory has a repository version of a package. (If you can't think of why you'd need to do that, you probably don't.)

The miniconfig.sh creation script got moved from soures/toys into more/, and the mini.config used to build the kernel tarball is now saved in the system image tarball.

November 5, 2015

Aboriginal Linux 1.4.3 is out, based on hg 1807, using Linux 4.1, toybox-0.6.1, musl-1.1.12, and busybox-1.24.1.

Our first nommu target, sh2eb, builds a system-image that boots on the Numato Mimas v2 board described on the jcore processor site (rather than under qemu). Toybox defconfig now builds nommu, we upgraded busybox versions to get some nommu fixes in hush, and there were several toolchain fixes since last time.

Multiple users requested bzip2 compression-side back (even though you can build it natively), and we added strndup() from posix-2008 to the last-ever release of uClibc (which we're migrating off of in favor of musl-libc, but our old Linux From Scratch 6.8 build control image refuses to build on a libc it doesn't specifically recognize so we need to upgrade the version of that to have a better test environment.

We added an /etc/os-release file which is what Debian now recommends for OS version identification now that they've given up on the Linux Foundation's "Linux Standard Base" bureaucracy. Also the initramfs should have a /bin/sh symlink pointing to hush (which is replaced with a link to bash by the dev-environment.sh init script, but if you run-emulator.sh without the native-compiler overlay it's nice to have).

September 13, 2015

Aboriginal Linux 1.4.2 is out, based on hg 1786, using Linux 4.0, toybox-0.6.0, and musl-1.1.11.

Haven't updated busybox in a while because we're slowly replacing it with toybox. The list of busybox commands still in use is:

awk bunzip2 dd diff expr fdisk ftpd ftpget ftpput gunzip gzip hush less pgrep ping pkill ps route sha512sum tar test tr unxz vi wget xzcat zcat

The new "sh2eb" and "sh2elf" targets produce working cross compilers for the jcore j2 open processor, and provide a start for general nommu support. Userspace is not yet cooperating, mostly because I'm only halfway done cleaning out MMU assumptions in toybox, but I hope to hae that working next release. (This release switched from ash to hush as the /bin/sh busybox provides because hush works on nommu and ash doesn't.)

The top level build.sh script now knows that if a target doesn't specify a KERNEL_PATH (where to grab the kernel binary out of the linux build), don't bother building anything but the cross compilers. This lets sh2elf stop early but still participate in buildall.

Switched the x86-64 target to build with musl-libc instead of uClibc by default. The x86-64 target broke when we switched the uClibc config from pthreads to NPTL in January, but only certain uses of it: the more/chroot-splice.sh build still works fine. The symptom is that distcc's attempts to call out to the cross compiler from within qemu hang, due to some NPTL bug in uClibc that's not worth fixing because uClibc is going away. (I'm aware of uClibc-ng, I'm also aware of ecommstation's attempts to keep OS/2 alive.) This problem is further compounded by QEMU's failure to emulate x86-64 floating point accurately, so the perl build breaks in the lfs-bootstrap build control image. Since this is already screwed up beyond easy fixing (other than reverting back to pthreads), I just switched it to musl because putting effort in making musl work isn't wasted the way it would be in uClibc. (The blocking issue with musl is that the old Linux From Scratch 6.8 build is full of packages with #if glibc/ #else if uClibc staircases that #error out at the end if they can't identify your C library. I need to update to a current LFS build in which those packages have hopefully been fixed.)

Also updated e2fsprogs build in host-tools.sh to a newer version. (It's not used by the build itself, but the dev-environment.sh script uses it make writeable scratch space images for qemu native builds.) This is because gcc 5.2 defaults to building in c99 mode, which breaks the older e2fsprogs, and that may start showing up in people's host linux distributions at some point.

May 6, 2015

Aboriginal Linux 1.4.1 is out, based on hg 1756, using Linux 3.19.

It's basically just the kernel version upgrade, still a release behind but no longer _two_ releases behind. Something broke the x86-64 lfs-bootstrap build under qemu (floating point rounding error in the perl build, which dies with ""Unable to find a perl 2.36242958346314e-310"), so I did a chroot version for now.

March 11, 2015

Aboriginal Linux 1.4.0 is out, based on hg 1748, using Linux 3.18 and toybox 0.5.2.

The simple root filesystem now lives in initmpfs, with the native compiler in squashfs on /dev/hda and spliced into the $PATH at runtime by the init script. (This also provides an example of how to do that on other systems.)

You'll need an updated build control image to automatically build things like Linux From Scratch which want to setup a chroot. (The old "should we copy our root filesystem into a chroot before modifying it" test was "is / writeable", the new test is "is /home bigger than /". The new root filesystem is writeable, but only a few dozen megabytes large.)

This release has switched to gzip tarballs instead of bzip2, due to the latter slowly becoming obsolete (as evidenced by kernel.org dropping use of the file format). The busybox implementation of the gzip compressor in insanely slow (slower than bzip2!) but it works and is scheduled for replacement by toybox.

Toybox is now handling sed, which means the remaining busybox commands which toybox still needs to replace are:

bunzip2, dd, ftpd, gunzip, pgrep, route, test, wget, diff, ftpget, gzip, ping, sh, tr, xzcat, bzcat, expr, ftpput, less, pkill, sha512sum, unxz, zcat, awk, fdisk, getent, man, ps, tar, vi

Of these 31 commands, toybox implements three (bunzip2, bzcat, and zcat) which busybox can't use due to busybox "tar" requiring then to be built in. Toybox' "pending" directory already has at least partial implementations of dd, gunzip, pgrep, route, test, diff, ftpget, gzip, ping, sh, tr, xzcat, expr, ftpput, pkill, fdisk, ps, and tar. This leaves ftpd, wget, less, sha512sum, awk, getent, man, and vi still to be implemented.

December 27, 2014

Aboriginal Linux 1.3.0 is out, based on hg 1715, using Linux 3.17 and toybox 0.5.1.

Patched out the bash function definition exports for that heartburn thing, fixed several hiccups in the RECORD_COMMANDS functionality (logging what the build actualy does while compiling stuff), broke x86-64 by enabling TLS (doesn't quite work reliably, it seems), worked around a busybox bug where sed -i on a nonexistent file would create it with random permissions (toybox sed, coming soon), wrestled with first nommu platform (not ready yet)...

The usual. Still catching up to the kernel releases.

September 26, 2014

Aboriginal Linux 1.2.9 is out, based on hg 1688, using Linux 3.16 and toybox 0.4.9.

Just the two package upgrades this time, I was still a kernel release behind and I didn't upgrade toybox to the release back in June. New version of each coming up in a week or so, so another catch-up release.

This also goes back to using busybox's cpio for now, because the linux from scratch build uses the as yet unimplemented cpio -s (in toybox source control, but not in the 0.4.9 release).

September 9, 2014

Aboriginal Linux 1.2.8 is out, based on hg 1684, using Linux 3.15 and a new ccwrap supporting musl-libc.

This is an interim release. An upgrade to linux 3.16 and a new toybox release are pending.

The big news is the new option to build musl-libc images with the new config knob MUSL=1, although the default is still uClibc for the moment because everything needs to be retested thoroughly, and not all targets are supported by musl yet.

Making even this much work required a from-scratch rewrite of ccwrap.c (which started life as the old uClibc compiler wrapper, before they gave up and started buildroot, and is thus full of uClibc-specific assumptions). This rewrite caused regressions galore, and just making uClibc work again with the new wrapper (for regression testing purposes) took a lot longer than expected. It's back to building Linux From Scratch 6.8 with uClibc again: next up is to build those same packages with musl. (And upgrade toybox, and to the 3.16 kernel...)

The new musl targets are still a bit... wobbly. Static linking against musl works, dynamic needs fixing. (It looks like -Bsymbolic-functions in the linker isn't always working yet.)

Other bugfixes: patch toybox not to build break when uClibc does something stupid with iconv.h. (New toybox release has a proper fix, but that's for next time.) SYSIMAGE_TYPE=initramfs is using the right image name again. The usr/src directory now has a hello.c (the README said it did, but only the C++ and threaded versions were actually there).

The sh4 kernel config is no longer forcing CONFIG_EXPERT (that patch should probably go upstream at some point, but after the number of years perl removal took it's hard to muster any enthusiasm for engaging with linux-kernel. Oh, and the qemu commit that broke signal handling in qemu 2.x was qemu git 02c4bdf1d2ca, I pinged qemu-devel about that but in case you want to revert locally and rebuild your qemu-system-sh4... It works fine for automated native builds either way, but interactive use is inconvenient when ctrl-c kills the _emulator_ instead of passing through to the shell instance.)

Speaking of qemu upgrades, two kernel patches went away: the sh4 one to not touch register 18 (qemu no longer exits over that), and qemu's powerpc board can now emulate 4 drives hooked up to the primary disk controller, so we don't need to use the cdrom controller to get an hdc, which means native builds don't need us to patch Linux's insane scsi device probing ("throw all the devices in a big pot and give it a stir, then grab one at random to be sda") to align qemu's idea of -hda, -hdb, and -hdc with what Linux hallucinates them to be. (Now there's only one controller and the Linux scsi guys are still working on screwing up device ordering _within_ a single controller.)

Bobby Bingham sent in a patch to stop yet another hardwired path stupidity where gcc was still reaching around ccwrap.c to screw up. (Rewriting the path to start with --nostdinc --nostdlib isn't quite _enough_ to get gcc to stop imagining paths to check; you have to patch out bits o the C code that try to grab random stuff, because gnu.)

Also, more/record-commands.sh infrastructure got upgraded to record the second half of the host-tools build (rebuilding the wrapper directory each time we add something to build/host, but in a way that _doesn't_ rebuild the directory if it already exists to avoid interfering with multiple parallel builds ala FORK=1 more/build.sh).

May 3, 2014

Aboriginal Linux 1.2.7 is out, based on hg 1649, using Linux 3.14, and toybox 0.4.8.

Overdue maintenance release to catch up to the kernel. Yes, we skipped a kernel release. And I meant to upgrade to busybox 1.22.1 but thought I already had because the last digit matched. And the musl switchover isn't finished yet. Yeah, my todo list still runneth over, and most of it's going to toybox.

At least the native stuff is building a new dropbear version. (Remember: the NSA didn't put the goto fail exploit in iOS, or the other goto fail exploit in gnutls, or the heartbleed bug into openssl, and I can't WAIT to see what they didn't do to dropbear.)

Tested under qemu 2.0, and in fact powerpc requires it because /dev/hdc moved. (Don't ask me why qemu-system-sh4 kills the emulator when you hit ctrl-c, it's been doing that for a while. They left the terminal in cooked mode and didn't intercept signals and I should probably poke them about that. It's on the todo list...)

November 19, 2013

Aboriginal Linux 1.2.6 is out, based on hg 1638, using Linux 3.12 and toybox 0.4.7.

There's a bugfix for hosts using patch 2.7.x, and creating sparse disk images for the native builds uses the "truncate" command now rather than dd. Gunter Roeck fixed a build regression if you want to build ext2 system images, and Patrick Lauer made the smoketest timeout configurable (and extended the default to 3 minutes).

(I'm way behind on email and distracted by other things. Hoping to get more done next release, but meantime here's an update to the new kernel. Tested against qemu 1.6.1, the usual suspects seem to work fine so far.)

September 17, 2013

Aboriginal Linux 1.2.5 is out, based on hg 1627 and using Linux for Workgroups (3.11) and toybox 0.4.6.

Same versions of the other packages as last time, even busybox and uClibc. The musl migration is coming but didn't quite make it this release. The busybox to toybox migration continues, with toybox now supplying grep and umount. This leaves busybox providing:

ash awk bunzip2 bzip2 cpio dd diff expr fdisk find ftpd ftpget ftpput gunzip gzip install less lspci man mount pgrep ping pkill ps route sed sh sha512sum tar test tr unxz vi wget xzcat zcat

(Several of which are already in the toybox "pending" directory defaulting to "n" while they await review and cleanup.)

If you use qemu 1.6.0, note that mips and sparc networking are broken. 1.5.0 works. No, I don't know what the qemu guys are using for regression testing either.

July 7, 2013

[UPDATE: missed checking in a file, hence the .1 release. Old broken tarball's been moved to old.]

Aboriginal Linux 1.2.4.1 is based on hg commit 1612, using linux 3.10, busybox 1.21.1, and toybox 0.4.5.

Took a while to work around QEMU's issues with arm, and sparc, and sh4, but it's got workarounds for all of them now. That means x86, x86_64, mips, mipsel, sparc, sh4, powerpc, and arm all work. (And sparc can actually do native builds now, under qemu 1.5.)

The mips64 target is bit-rotted (I think qemu changed out from under it somewhere between 1.0 and 1.2), and m68k support in qemu still isn't finished. Todo items.

Meanwhile, toybox replaced 4 more commands in the build (mv, stat, ifconfig, and split). Upcoming work: I've done part of the architecture to put simple-root-filesystem in initramfs (or initmpfs) always and splice the native-compiler in at runtime from /dev/hda with symlinks, that might be ready next release. And I'm chipping away at the switch to musl-libc.org (basically a complete rewrite of sources/toys/ccwrap.c to _not_ assume uClibc).

Binaries in the usual places.

April 2, 2013

A long over due release: 1.2.3 is based on hg commit 1589, using busybox 1.21.0, toybox 0.4.4, and linux 3.8.

The delay is because qemu 1.3.0 and 1.4.0 both have a subtle bug causing intermittent instability in the native build on all targets, which I blamed on everything from toybox to the kernel until working out that if I went back _two_ releases to qemu 1.2.0, Linux From Scratch built reliably on the various targets. (A bug that goes away when you try again is always a pain to track down, you're never quite sure whether you've fixed it or just didn't see it this time...)

The about page got a major facelift, if you haven't noticed you might want to give a look.

December 16, 2012

Another kernel, another release: 1.2.2 is out based on hg commit 1576. It's got Linux 3.7 and toybox 0.4.2.

The native-built binaries for dropbear and strace and such are back. (Last time around native building couldn't upload binaries to the host because the launcher script was calling busybox netcat to host ftpd, but toybox is providing netcat now. I punted because we were close to the next kernel release and I wanted to catch up.)

Other than the package upgrades, the rest is pretty minor. The screenshots directory is back. Undid some minor bit-rot in the i686-virtio target, and a tweak to powerpc so musl can build natively on it. (Haven't switched over any targets to build musl on the host yet, still chipping away at a ccwrap rewrite.) The chroot-splice.sh script is now doing its bind mounts read only.

November 15, 2012

The 1.2.1 release is out, based on hg commit 1560, with linux 3.6 (plus some ext4 bugfixes), busybox 1.20.2, and a uClibc config tweak so mdadm can auto-scan arrays. Zlib is now at 1.2.7 (used by squashfs in host-tools). Toybox and binutils also got updates, as described below.

Toybox was upgraded to 0.4.1 and the build is now using it by default. The goal is to eventually replace busybox with toybox, and each release from here on is likely to switch over a few more commands. The current list is:

Busybox:

ash awk bunzip2 bzip2 cp cpio cut dd diff dnsdomainname egrep expr fgrep find ftpd ftpget ftpput grep gunzip gzip ifconfig init install less losetup lspci man mount mv pgrep ping pkill ps readlink rm route sed sh split stat tar test time touch tr umount unxz vi wget xzcat zcat

Toybox:

basename bzcat cal cat catv chgrp chmod chown chroot chvt cksum clear cmp comm count date df dirname dmesg dos2unix du echo env false free head help hostname id insmod kill killall link ln login logname ls lsmod md5sum mkdir mkfifo mknod mkswap mktemp modinfo mountpoint nc netcat nice nohup od oneit passwd patch pidof printenv pwd realpath rmdir rmmod seq setsid sha1sum sleep sort swapoff swapon switch_root sync tac tail taskset tee true truncate tty uname uniq unix2dos unlink unshare uptime usleep vconfig vmstat w wc which whoami xargs yes

Binutils has been upgraded to the last GPLv2 version in the git repo (commit 397a64b3, between 2.17 and 2.18), which has the -Bsymbolic-functions command required to build the musl C library. (This version of binutils also claims to have ARMv7 support, but we're not using that yet.)

The goal for next release is to replace uClibc with musl for the architectures musl currently supports, and to help the musl developers add support for the remaining architectures to eventually replace uClibc as a cross-compiled package. (New build control images to natively build uClibc and glibc chroot environments on an arbitrary target are on the todo list. Supporting a libc does not require cross compiling it.)

On the infrastructure side of things, USE_ALT went away. The new way to build alternate versions of packages is to just provide a directory of extracted source in the "packages" directory, in which case the build will use it verbatim (applying patches is your problem).

This means the following works:

more/clean.sh
git clone git://busybox.net/busybox packages/busybox
./build.sh i686

If you want to go back to building the tarball versions, either set IGNORE_REPOS=busybox (or all), or "rm -rf packages/busybox".

August 22, 2012

After far too long the 1.2.0 release is out, based on hg commit 1535. Package upgrades are Linux 3.5, uClibc-0.9.33.2, busybox 1.20.0, toybox is back in at version 0.4.0, and most targets are finally using NPTL instead of pthreads.

Several "how did that ever work" bugs in the build scripts got fixed (commits 1513 through 1518, for example). Several other "only occurs on a specific distro" bugs also got fixed (a Gentoo one in 1518, a Fedora one in 1532). The longstanding uClibc++ build break where it complained ".config is a directory" was squashed by 1533.

The new development was mostly prep work for toybox integration and fixing bit-rot in record-commands.sh (used to test toybox). There was also a lot of dealing with regressions introduced by package upgrades. (Upgrading uClibc broke stuff. Upgrading busybox broke stuff. Upgrading the kernel broke stuff all three times.)

Busybox is still doing most of the heavy lifting both during the host build and on the target, but the baseconfig-busybox file is now sorted with toybox-supplied functionality at the end, allowing toybox to replace busybox code incrementally as commands become ready.

In future versions I'm looking at migrating this project from busybox/uClibc to toybox/musl, producing something compatible with Android's "no GPL in userspace". I'm also looking at replacement toolchains, since it may take me a while to clear time to work on qcc. If you're curious about all that, ask on the mailing list.

January 31, 2012

The 1.1.1 release is out, based on hg commit 1496.

The new release upgrades packages (Linux 3.2, uClibc 0.9.32.1, busybox 1.19.3, e2fsprogs 1.41.14), and natively compiles Linux From Scratch 6.8 (which involved adding mkostemp() to uClibc, for util-linux).

The x86 targets now require QEMU 1.0 (because the "qemu" binary got renamed "qemu-system-i386"), but the mips targets require the previous release due to a regression in the network card emulation (fixed in the source repo, but no 1.0.1 is out yet).

On the documentation front, the 2008 "what are we doing and why" presentation is now available in HTML format, probably worth reading if you haven't yet. The FAQ also grew some new entries (showing how to use more/test.sh to rebuild uClibc without redoing a while toolchain, and why powerpc isn't as backward compatable as you'd expect).

The long-broken sparc target now supports basic dynamic linking (although dynamically linked threading and C++ are still broken).

Busybox is back to using a specified configuration instead of just switching on everything. (Busybox's new ubifs commands won't build on Ubuntu 11.10, and it's hard to see the command count grow past 300 and keep going without humming the "Katamari Damacy" theme.) If we missed a command let us know, and you can of course still build busybox defconfig natively. (Eventually, we'll start using toybox again, which has been relaunched as a BSD-licensed project.)

The build.sh wrapper can now rebuild from any stage, just specify the starting point on the command line after the architecture, for example "./build.sh i686 simple-root-filesystem" won't rebuild the cross or native compilers, or the kernel, but will rebuild the root-filesystem, root-image, and system-image stages.

The more/test.sh script now understands "host" as an architecture, for testing host-tools builds.

Rename the USE_UNSTABLE variable to USE_ALT since it already uses alt- prefixes on the packages and patches. The repository's build/patches directory no longer contains "alt-*.patch" symlinks, instead a new more/alt-patches.sh script can set them up locally for a given package.

Add record-commands wrapper to target image, "record-commands /mnt/init" runs /mnt/init and logs commands to "/tmp/record-commands-log.txt". This way you can see what commands target builds are running.

The CPUS calculation, for automatically parallelizing the build on SMP systems, now detects hyper-threading and won't stress hyper-threaded systems quite so hard as full SMP. The target's sbin/init.sh also has basic CPUS detection code now too, since KVM or a chroot can have more than one. (This only matters if CPUS isn't set by something else, such as the dev-environment.sh wrapper script's distcc setup.)

Several bugfixes, due to insufficient regression testing of previous releases: SYSIMAGE_TYPE=ext2 SYSIMAGE_HDA_MEGS was broken, more/smoketest-all.sh was looking for static-tools output in the old place, and more/record-command.sh only worked with NO_SANITIZE_ENVIRONMENT=1. The target root filesystem's init script wasn't detecting a failure to mount devtmpfs (such as when used to setup a chroot on a kernel that didn't support it), and falling back to mdev -s. And ldd was spewing debug messages. All fixed now.

October 2, 2011

The big news is that Linux From Scratch 6.7 now builds to completion under 11 targets, via the lfs-bootstrap build control image. The remaining platforms all show target-specific issues (often just insufficient board support in QEMU) rather than generic problems with the build environment. The big bugfix allowing this to happen was the m4 build hang (which was a bash bug).

The only upgraded package was BusyBox 1.19.2, but there were several performance tweaks and bugfixes, such as:

• The more/chroot-splice.sh script sets up a chroot using the build/root-filesystem-$TARGET directory and a build control image, for quickly testing build control images natively (generally using the i686 or x86-64 root filesystems).

• The more/buildall-native.sh timeout value is now configurable via the TIMEOUT=seconds environment variable (defaulting to 60).

• Tuned the more/buildall.sh script in FORK=1 mode to take available memory into account determinging CPUS (so it doesn't overload the build machine so easily).

• When dev-environment.sh enables distcc it sets CPUS=3 because several current QEMU boards support a maximum of 256 megs physical memory.

• The ldd and ldconfig binaries in the cross compilers were target binaries instead of host binaries: that's fixed now.

• The virtio targets should now work with dev-environment.sh

• The test script more/test.sh should no longer delete the output directory (build/$STAGE_NAME-$TARGET) at the start of each test, allowing test rebuilds of things like uClibc which have target prerequisites.

• The common kernel config now uses the ext4 driver for both ext3 and ext2 filesystems.

August 22, 2011

The 1.0.3 release is out, based on hg commit 1430. This release includes the long-awaited Linux 3.0 and uClibc 0.9.32 releases, along with BusyBox 1.19.0 (and Dropbear 0.53.1 in the static tools).

This release is still using pthreads, but NTPL support is a goal for next release. (It's just a uClibc config change.)

The other big news is that the control-images got split out to their own repository, with their own web page.

The Aboriginal Linux project is feature complete ever since the 1.0 release, and other than adding support for more targets the only major todo items left are a big documentation update, more automation via cron job, and enabling NPTL support. (This release is still using pthreads, but NPTL support is a goal for next release. It's just a uClibc config change at this point, 0.9.32 has support but needs more tweaking and testing.)

Other changes:

• The default kernel configuration replaces includes v9fs (as used in virtfs or the diod server) instead of the older NFS. In addition Alessio Igor Bogani contributed i686-virtio and powerpc-virtio targets, using the new virtio infrastructure in KVM to provide efficient emulation-friendly virtual devices.

• Several random bugfixes and cleanups (BINARY_PACKAGE_TARBALLS should work again, the root filesystems now use devtmpfs instead of mdev).

• Major cleanup on sources/targets, now most entries are a single file. The hold hw- prefix stuff went away, now the ones that are files are built by more/buildall.sh and the ones that are directories are the derived stuff that only gets built if you specify it explicitly. (read_arch_dir is now called load_target, and the new base_architecture function reads in a base architecture that this one derives from.)

• The new more/smoketest-report.sh produces CSV output with more info. (I need to follow this up with a nightly cron job and a pretty web page generator.)

June 14, 2011

The long overdue 1.0.2 release is out, based on hg commit 1375.

This release includes several bugfixes, package version upgrades for Linux (2.6.39), BusyBox (1.18.4), and Squashfs (4.2), and some minor user interface tweaks. (Still using uClibc 0.9.31, NTPL is bumped to next release.)

Targets

This moves forward three(!) kernel releases (sorry), and although several targets bit-rotted along the way (notably armv6l, sh4, and mips) they're all fixed now, including some that didn't work last release.

QEMU still doesn't emulate the big endian arm or m68k targets, powerpc-440fp needs to be switched to QEMU's -M bamboo board emulation now that it can actually emulate a 440, and mips64 and sparc boot to a shell prompt but contain bugs in either uClibc or the kernel that make them unreliable.

All the other targets can at least natively compile and run "hello world", and the arm, x86, mips, and powerpc targets built static versions of dropbear and strace (available in the extras directory, or by using native-build.sh with the static-tools.hdc control image).

Bugfixes

• Commit 1323 fixes a bad path in the static-tools build)
• Commit 1327 fixes ccwrap to understand -xc
• Commit 1328 puts libnsl back
• Commit 1332 makes #!/bin/sh point to bash instead of hush
• Commit 1335 makes download.sh check if sha1sum is installed on the host (so it can fail with a better error message)
• Commit 1344 serializes scratch disk image creation during parallel builds of all targets (so the disk I/O storm doesn't trigger timeouts during native builds of dropbear and strace).
• Commit 1364 upgrades busybox to fix a wget hang.

New Control Knobs

Two new build stages root-image.sh and linux-kernel.sh got split out from system-image.sh. The first makes a filesystem image from a root filesystem directory, the second builds a kernel. This leaves system-image.sh to create the qemu launch scripts and bundle it all up into a tarball.

The advantage of the split is you can repackage the root filesystem without rebuilding your kernel. (Unless you chose SYSIMAGE_TYPE=initramfs, where the output of root-image.sh is a prerequisite to linux-kernel.sh.) The downside is you have to remember to call the other scripts before calling system-image.sh in order for your changes to wind up in the system image. (If you delete the tarball you modified out of the build directory, ./build.sh should still work out what it needs to rebuild.)

Environment variables: SYSIMAGE_HDA_MEGS now defaults to 2 gigabytes (only currently matters for SYSIMAGE_TYPE=ext2). The more/buildall.sh script now pays attention to NO_CLEAN and BUILD_NATIVE_ONLY. As always, the environment variables are documented in the config file at the top of the source tree, and leaving them blank should provide sane default behavior.

Two new scripts in the system image's /sbin directory are:

• setup-chroot /home/mydir [COMMAND...]

  This script copies the system-image's (generally read-only) root filesystem into a writeable subdirectory, copies the other mounts ("mount --bind /proc /home/mydir/proc" and so on), and chroots into the resulting directory to run the rest of the command line. (If no commands are specified, chroot runs /bin/sh for an interactive shell prompt.)

  When the chroot exits, the script cleans up all the mount points but leaves the directory contents.

  This lets you use a read-only system image to build and install software into a chroot.

  If the target directory already exists, the script will just redo the --bind mounts, and won't copy files.

• zap-chroot - The cleanup script to remove all mounts under a specified subdirectory.

In the system images, root's home directory is now /home/root so it's in known writeable space you can stick a .ssh directory in for dropbear.

February 13, 2011

February 8, 2011

The website was down again while we switched web hosts, but we're back up on dreamhost now, which has buckets more bandwidth and lets us have a mailing list again.

I still need to figure out how to enable mercurial on the new host, so the repository is down at the moment, but I should get that fixed up soon.

January 2, 2011

It's a month late, but Aboriginal Linux 1.0.1 is finally out, based on hg commit 1318, using Linux 2.6.36, uClibc 0.9.31, and BusyBox 1.18.0.

The big news this time is the native build control images, especially the one to build Linux From Scratch 6.7.

These control images drive the automated build infrastructure (first introduced back around release 0.9.8 and substantially improved since then). They allow you to fire up a system image under QEMU and run a completely automated native build.

To use a control image, cd into a system-image directory and run "./native-build.sh path/to/control-image.hdc". The result should wind up in the "uploads" subdirectory. To build control images from source, run "more/build-control-images.sh" and look in "build/control-images" afterwards. The build scripts live in "sources/control-images".

The lfs-bootstrap.hdc image builds most of Linux From Scratch 6.7. It doesn't replace the toolchain or C library, and stubs out internationalization support, but builds everything else (about 50 packages total). This build runs to completion reliably on i686, x86_64, and armv5l. Making it do that smoked out a lot of subtle bugs and configuration issues in the build environment, resulting in much improved system images. This LFS build sometimes finishes on mips and powerpc, and sometimes randomly dies for various asynchronous reasons that seem like something wrong in uClibc's pthreads locking code, although that's just a guess. (Since the uClibc developers (finally!) shipped a 0.9.32-rc1 with NPTL support on all targets, I expect the fix will involve upgrading. Part of the delay in this release was me playing around with that, but I'm releasing a checkpoint of the existing code before switching over to the new threading implementation.)

I fixed the static "dropbear" binary to call "ssh" out of $PATH instead of expecting dbclient at a fixed location, fixed the guest account's /etc/passwd and /etc/group entries so dropbear can use it, and added a /root directory to the system images so you can add a .ssh directory to it to make dropbear happy (possibly by mounting a tmpfs on it first).

Build infrastructure tweaks: sources/download_functions.sh is now its own file, and various fixes (only apply patches that end in ".patch") . The root filesystem's sbin/init.sh now mounts /dev/pts. Bugfix for SIMPLE_ROOT_OVERLAY, and multiple bugfixes in ccwrap.

Lots of tweaks to the native-build stuff. Redo control image generation, add bootstrap-skeleton with common code for bootstrapping distros (it has its own README), and rebase the half-finished gentoo-bootstrap on top of that. Upgrade "bisectinate" debugging script to take its test from the command line.

Several tweaks to uClibc to build LFS packages, and tweaks to the base kernel config to support NFSv3 and devtmpfs.

The new more/chroot-splice.sh script combines a root filesystem directory and a control image directory into a combined version you can chroot into and run a build from. (It looks for each in the location the build puts them, build/root-filesystem-$ARCH and build/control-images/$IMAGENAME respectively.) The new more/zapchroot.sh script unmounts everything under a directory, which is useful when playing with a root-filesystem via chroot.

I've fallen a bit behind on regression testing the targets, and hope to catch up next release. (Upgrading the kernel and QEMU breaks things that used to work.)

The i486, i586, i686, x86_64, powerpc, armv4l, armv4tl, and armv5l targets all work (for me) without qualification.

Last I checked, qemu still didn't support armv4eb or m68k, so I can't test those, but they build. (The kernel .config is almost certainly wrong for any target you'll actually be able to run them on, but the toolchain and root filesystem are probably ok.)

Updating the kernel broke armv6l: attempting to boot it under qemu exits with an "unimplemented cp15 register write" error. I'm not sure whether this is a kernel bug, a qemu bug the upgrade triggered, or some kind of misconfiguration. The proper fix is probably to look at the other arm boards QEMU can emulate for newer processors, and switch to one of those.

QEMU had a MIPS bug that prevented the system images from working for a version or two (such as the one in Ubuntu 10.04). It's fixed in current qemu-git, and isn't actually a problem with the mips, mipsel, or mips64 system images, they work fine under older or newer versions of qemu.

Sparc broke again (failure to launch init), and Oracle's patent trolling makes it harder than ever to care. It's on the todo list...

The only system image that doesn't current compile is sh4. (As noted in the 0.9.11 release notes, the sh4 maintainer broke it on purpose, which also makes it hard for me to care much about that target.)

The problem with sh4 is once again the kernel, this time a build break:

arch/sh/kernel/process_32.c:303: error: conflicting types for 'sys_execve'
/home/landley/play/two/build/temp-sh4/linux/arch/sh/include/asm/syscalls_32.h:24: error: previous declaration of 'sys_execve' was here

And even if I bothered to fix that, another change in the sh4 code changed the serial devices so QEMU's serial console no longer produces output. (Note that the sh4 test image on qemu.org uses a 2.6.29 kernel, which may be the last time the thing actually worked for anybody who isn't in a financial relationship with Renesas.)

Last I checked, QEMU still didn't support armv4eb or m68k, so I can't test those. I have patches to make m68k work on aranym and armv6eb work on actual big endian arm hardware, but haven't merged them yet.

November 3, 2010

Still working on the next release, currently scheduled for around the start of December. Until we get a new mailing list, the source control log shows what's changing, and you can always email me directly, rob at landley dot net.

September 5, 2010

Aboriginal Linux 1.0 is out, based on hg commit 1238, using Linux 2.6.35, uClibc 0.9.31, and BusyBox 1.17.2.

Yes, it's been over five months since the last release. I didn't want to ship yet another intermediate release before doing 1.0, but I was trying to run out of todo items before shipping. (Yeah, I should know better. Sorry for the delay.)

The following are release notes, not proper documentation. Proper documentation (including an introduction to the whole thing) is the next big TODO item.

The arm, mips, powerpc, and x86 targets all work. Sparc works statically linked, but dynamic linking is still broken. Khem Raj fixed mips64 dynamic linking, but the native compiler still doesn't work. The m68k target builds again, although QEMU still doesn't emulate it. The sh4 target and big endian arm are are still broken.

The snapshots page needs updating. The documentation needs updating.

The native-build.sh script moved into each system image. This means you can now download a system image tarball and a control image, then run a native build without any dependencies on the Aboriginal Linux build scripts. This script is a wrapper around dev-environment.sh which takes one argument, the name of the control image file.

The build only puts dev-environment.sh and run-from-build.sh into a system image when the system image includes native development tools.

This release adds prebuilt binary build control images, generated by more/build-control-images.sh and shipped in the control-images directory. These are squashfs filesystems containing source code and an init script to build it and upload it to the host (using the $FTP_SERVER and $FTP_PORT variables, set by native-build.sh).

If these images are loaded into /dev/hdc of the emulated system, the system image's init script will mount that on /mnt and run /mnt/init instead of a shell prompt, so the build is completely automated.

Control images are provided to build hello world, build static dropbear and strace binaries, run the busybox test suite, and an in-progress script to bootstrapping Gentoo Stage 1 does about the first 2/3 of the job.

The root filesystem broke into two stages: simple-root-filesystem just contains a simple "defconfig busybox plus uClibc" system, enough to boot to a shell prompt. The native-compiler stage is like cross-compiler only unprefixed, built to run on the target, and including make, bash, and distcc. The root-filesystem stage combines the two into a single root filesystem with native development tools.

If the root filesystem is mounted read only, the init script will now mount a tmpfs on /home if there's no /dev/hdb to provide writeable scratch space. Chroot environments should now use Google's 8.8.8.8 nameserver by default. The distcc binaries moved out of /bin to the /distcc directory.

The "oneit" binary became a standalone program, and uses the / directory as PID 1's working directory, allowing you to umount /home if necessary. Toybox's "patch" command went upstream into busybox, and busybox's defconfig once again supports netcat server mode, so toybox was removed. We're now using a defconfig busybox, so configuring it properly is their problem.

The kernel configuration got collated to sources/baseconfig-linux (just like baseconfig-uClibc), and the various packages were migrated to use that. The kernel config is now the baseconfig plus the settings entry $LINUX_CONFIG. You can still supply a miniconfig-linux file to override this, and some architectures still do that. The ones that have been migrated to use a shared baseconfig are a lot more regular now, and the differences between architectures broken out and isolated.

The build stages were broken up to be more granular. The root-filesystem.sh stage now merely collates the output of two previous stages: simple-root-filesystem.sh (which produces just enough of a root filesystem to boot to a shell prompt, based on uClibc and busybox plus a few init scripts and config files), and native-compiler.sh (a portable native toolchain).

The system-image.sh stage now generates the dev-environment.sh script. It only puts dev-environment.sh and native-build.sh scripts in the tarball when NO_NATIVE_COMPILER is blank (and thus the build includes native development tools), or if root-filesystem isn't available in which case it'll automatically use simple-root-filesystem.

Several more package builds were moved to sources/sections.

The new "more" subdirectory collects together all the user-runnable scripts that aren't build stages. (Many of these were previously in sources/more.) New scripts in this directory include:

• Scripts to run newly built system images out of the build directory ("run-emulator-from-build.sh $ARCH", "dev-environment-from-build.sh $ARCH", and "native-build-from-build.sh")
• test.sh - Wrapper script to run commands in the build environment, with read_arch_dir loaded. Useful for things like "STAGE_NAME=simple-cross-compiler more/test.sh sparc build_stage uClibc", or to dump the build context's environment variables ala "more/test.sh env"
• timeout.sh $SECONDS command... - Wrapper to run a child process and kill it if it doesn't produce a line of output for $SECONDS seconds.
• build-control-images.sh - build all control images from sources/native-builds.
• for-each-target.sh - iterate through every defined target architecture and run a command.

A new cron job is populating the snapshots directory. Currently only the stable versions, but the unstable versions should wander back in eventually. (When the bisectinate and migrate-kernel scripts catch up to what they need to do to properly test unstable packages.)

More documentation: The "about" page got redone, and the FAQ was fluffed out with a new debugging section. Michael S. Zick wrote up mips docs for the architecture page, and the build scripts grew even more comments.

The control image generation was upgraded so each control image source can have its own patch directory and file directory. (This needs more work.)

Environment variable sanitization: the sanitize_environment function unsets any environment variable it doesn't recognize, either from a whitelist or from config. Clearing all unrecognized environment variables insulates the build from some of the strange environments out there. To disable this behavior, export NO_SANITIZE_ENVIRONMENT=1.

New config variables:

• STATIC_CC_HOST became CROSS_HOST_ARCH, set that and second stage cross compiler gets built.
• The new ALLOW_PATCH_FAILURE=1 variable continues despite a patch failing to apply.
• NO_HOST_TOOLS=1 tells build.sh to skip the host tools step, which is seldom a good idea but there you have it. The new HOST_EXTRA is an equally horrible idea, but if you feel the need it's a list of extra commands to pass through from the host $PATH.

• NO_CPLUSPLUS=1 tells it not to add C++ support in the toolchains it builds.

• NO_CLEANUP isn't a new variable, but you can now set this for entire builds and it should work.

ccwrap improvements: Handle unexported $PATH better. It doesn't need to search $PATH when argc[0] is a path (either absolute or relative to the current directory). Also use Scrt1.o when building a shared library. (Oddly this never caused a problem before, but do the right thing anyway.)

Upgrades to BUILD_STATIC: it now takes a comma separated list of packages, with the special value "all" to build everything static, and "none" to build nothing static. The default is to build static versions of busybox and the toolchain packages (gcc, binutils, make, and ccwrap) both due to the increased performance this provides under qemu (which has to repeatedly re-translate executable code pages that modify themselves, such as when dynamic linking patches per-process jump addresses), and also due to the increased portability this provides the native compiler tarball.

(The host-tools stage always builds dynamic packages, because the glibc maintainer has a personal dislike of static linking, meaning glibc's static linking support (in addition to creating 400k "hello world" binaries) is both broken[LINK] and deprecated[LINK]. Some distros have even taken to breaking out the *.a format libraries used for static linking into a separate package, and not installing them by default. The host-tools stage now tests for this, and refuses to continue in a dynamic-only build environment unless you set BUILD_STATIC=none. You can rebuild under the resulting target system to natively compile static binaries, which uClibc supports quite well, but you can't generate them from the host.)

The old FROM_ARCH is now HOST_ARCH, and the infrastructure using it (cross-compiler.sh and native-compiler.sh) has been cleaned up. The HOST_UTILS variable was also removed, that's now built for the full cross and native compilers, but not the simple cross compiler.

The more/smoketest.sh file no longer creates hdb.img.

The host-tools stage has been upgraded to work around the utter stupidity of Ubuntu 10.04, which turned "gcc" into a perl script. (Yes really. No, I don't know what they were smoking.)

Build zlib in host-tools (for squashfs) and in static-tools for dropbear's -C. This is necessary for reproducing the build under a system-image, because the default system image type (squashfs) requires zlib to build.

Update the command prompt in system images to show the amount of parallelism available via distcc. (When it says 1 you haven't got distcc, and when it says a number greater than that you have.) The prompt now looks like:

(powerpc:4) /home #

The make, bash, and distcc builds are now part of the native-compiler stage (and their build scripts moved to sources/sections). This means that a native toolchain now includes make, because it's hard to build make without make. This stage also records the uClibc and linux config files for posterity.

The build was weaned off of toybox. (Broke out oneit, sent patch upstream to busybox, and got netcat fixed in defconfig so native-build.sh could use it.)

Several small bugfixes and cosmetic cleanups. For example, we now delete empty build/temp-* directories when we're done with them, and several config options such as NO_NATIVE_COMPILER and NO_CLEANUP should be more reliable now. Don't build hello-dynamic on a BUILD_STATIC=all system. Clean up debris files the kernel's headers_install leaves lying around. Minor upgrades to FORK=1 support, and QUIET is now part of FORK instead of a separate control. Generating a package manifest should no longer produce error messages. Empty temp directories get deleted on the way out. Better support for building older tools as alt-packages (tested binutils 2.14 and gcc 3.4). Comment out default UNSTABLE= URLs in download.sh, supply your own if you want that. Binutils build now honors existing $CFLAGS (if any). The dienow error message goes to stderr. More reliably track package cache state when patches fail to apply. Simplify system-image build (don't try to package root filesystem and build kernel in parallel anymore, squashfs and make both have internal SMP support). BUILD_VERBOSE is now more verbose. Reliability improvement for record-commands.sh. New shell functions blank_workdir and extract_package. Consistently use name-name in scripts and name_name in functions.

It's been a long 5 months...

May 2, 2010

Some reasons to move away from the old name are listed here, and some reasons to pick the new name are covered in this thread.

Some links may be broken while we switch over. Let us know what we've missed on the mailing list...

March 29, 2010

Version 0.9.11 is out, based on hg commit 1020. We're closing in on a 1.0 release, but not quite there yet.

This version upgrades to Linux 2.6.33, uClibc 0.9.30.3, and BusyBox 1.16.1.

Most targets now have prebuilt binaries for dropbear, strace, busybox. These are statically linked against uClibc, so just download, set the executable bit, and run.

These dropbear and strace binaries are built using the automated three-disk native build capability introduced back in November, which has been improved and genericized this release. Now one script creates a build image (see examples in sources/native-builds) and another script launches a system image with that build image attached as the third hard drive (/dev/hdc), telling the system image to automatically build those source packages and upload the result to the host via ftp.

So to reproduce the dropbear and strace binaries for a supported ARCH, you could do:

./build.sh ARCH
sources/native-builds/static-tools.sh tools.hdc
sources/more/native-build.sh ARCH tools.hdc output

When the build exits, the results should be in the host's "output" directory.

The arm, mips, x86, x86-64 targets that worked last time still work. The armv4eb and m68k targets still don't work.

The powerpc and sparc targets have been upgraded. Powerpc now works with the 3-disk automated native build (although you'll need qemu 0.12.4 to take advantage of this). The sparc dynamic linker still doesn't work in uClibc, but the statically linked system is now working well enough to build packages natively.

New targets this release are i486 (thanks David Seikel) and mips64 (which mostly works, but the native toolchain isn't quite properly configured to build stuff with yet).

Finally, the sh4 target was broken by a cosmetic cleanup of the startup assembly code in Linux 2.6.33 that rendered any binutils version older than 4 months unusable to build sh4. While it's possible to patch around this, it hasn't yet been worth effort. The Linux-sh4 maintainer is aware of the issue, but explicitly doesn't care about users other than his employer, and so far nobody else seems to care enough about supporting the discontinued Sega Dreamcast game console to argue with him. (Might revisit this next release, but m68k support is a higher priority.)

The system-image's "run-emulator.sh" is now a much smaller and simpler script, consisting primarily of the qemu command line. (And if you export DEBUG=1 it prints the emulator command line.) The distcc and hdb setup code moved to dev-environment.sh.

The screenshots page is working again, if a bit stale.

The bisectinate.sh script (to automatically git bisect a build issue) has been updated, more to come next release.

The download.sh script has been updated to understand FORK=1 and download packages in parallel. (And to extract them in parallel when EXTRACT_ALL=1.)

The new export_if_blank shell function allows several more environment variables to be overridden, including TOP, SOURCES, SRCDIR, PATCHDIR, BUILD, SRCTREE, HOSTTOOLS, WRAPDIR, CC, and STAGE_NAME.

I'd also like to thank the debugging efforts of Wolfgang Denk, who hit as many bugs trying to use this project as I normally do with other people's. He eventually got discouraged and left, but thanks to his efforts:

• We now unset several environment variables (such as ARCH and CROSS_COMPILE) that break things if the host exports random data in them before the build. (Yes, this fights with export_if_blank above, and the way configuration works in general. Some more design works is necessary here to let you configure the project with environment variables, but not be confused by existing environment debris.)

• BUILD_STATIC=none can how handle the case where the host toolchain can't build any static binaries at all.

• The CROSS_SMOKE_TEST implementation (which had bit rotted) was fixed and moved to its own script (sources/more/cross-smoke-test.sh), which is called by build.sh as necessary.

  Note that actually trying to use it is a question of getting qemu application emulation to work for your target. For arm you need to tweak /proc/sys/vm/mmap_min_addr as root. For ppc you need this patch. For mips64 somebody still needs to implement it (no matter what gentoo thinks).

  (There's a reason the smoke test was disabled long enough to bit-rot. It broke for qemu reasons more often than it found bugs in the toolchain.)

February 2, 2010

Version 0.9.10 is out, based on hg commit 966.

Yeah, I know 1.0 is overdue for a release, here's a resync point with the 2.6.32 kernel and uClibc 0.9.30.2.

There are some regressions this time around, which I plan to fix up in the next release. The 2.6.32 kernel broke strace, and upgrading to the new version didn't fix it. (I know how, I just haven't yet.) The m68k target isn't currently building due to the uClibc upgrade (yeah, "bugfix only", I know), and the armv6l target seems to be broken two different ways. I screwed up the screenshots page (which is halfway converted to Vladimir's new fancy version with style sheets, but "halfway" doesn't actually display right). The cron job building nightly snapshots got taken down when the server's hard drive filled up, and now it's in the process of being upgraded with a couple new scripts:

The new sources/more/migrate_kernel.sh script does a "make oldconfig" between the stable and "USE_UNSTABLE=linux" kernels, and creates a new alt-miniconfig-linux for the specified target with any new symbols that showed up. (This means the snapshots should break less, but adding the symbols to the native configs will still require manual filtering.)

Also, the new sources/more/bisectinate.sh script automatically does a git bisect between a known good and known bad (generally HEAD) version of one of the unstable packages (linux, uClibc, or busybox). It can test that the build completes, or that the system image boots and successfully builds the static native dropbear binary. (In theory this will allow the nightly snapshots to automatically isolate and report the first commit that broke them.)

In other general infrastructure news: some extensive upgrades to the compiler wrapper (ccwrap), such as no longer needing GIMME_AN_S at compile time and instead autodetecting the presence of libgcc_s.so vs libgcc.a and behaving appropriately. A largish cleanup/refactoring of the compiler build (described in commit 944) added a new "native-compiler.sh" script, which handles building the improved "cross-compiler" tarball that includes thread support and uClibc++ and is statically linked against uClibc on the host. The simple cross compiler is now "simple-cross-compiler.sh" (which is good enough to build a system image, but isn't very useful in other contexts).

The host-tools.sh stage now only builds e2fsprogs or squashfs if the build is actually going to use them (based on the SYSIMAGE_TYPE config symbol). The sources/sections scripts now come in two flavors: an *.build version with setup/cleanup called automatically and *.sh files which do their own setup/cleanup within the script. Each system image's run-emulator.sh will once again add /sbin:/usr/sbin to the $PATH if it can't find mke2fs.

The --extract command line option went away from download.sh because everything else works via environment variables. Use EXTRACT_ALL=1 instead.

Fixed a few regressions that snuck into previous versions: the config-{linux,uClibc,busybox} should once again be in the root filesystem's usr/src directory, buildall.sh should now try to build the static native dropbear/strace binaries for all targets (not just the last one) and it should properly be running smoketest.sh again.

December 8, 2009

Version 0.9.9 is out, based on hg commit 921.

Just a checkpoint on the way to 1.0, which is still planned for around new years, but there's been some schedule slippage already.

• New documentation

  The presentation slides have been converted to HTML, although the much prettier (and much bigger) PDF is still available.

  There is also a FAQ, which is still somewhat sparse. (Ask away.)

• Prebuilt binaries

Added shm support to uClibc. Bugfix to the c++ compiler (it should work again). The system images now have a "guest" user/group so packages can more easily be built as a non-root user. The shell prompt now includes the $HOST name.

• Build scripts

  A new trivial-but-convenient "./clean.sh" script lets you delete just the target builds out of "build". (If you delete that whole directory it has to re-extract all the package tarballs and rebuilt the host tools, which is often unnecessary work.)

  Command recording has been completely redone. No more RECORD_COMMANDS variable, instead run sources/more/record-commands.sh to wrap the current $PATH in build/wrappy (either after or instead of running host-tools.sh), then run your build stages, and finally either run sources/more/report-recorded-commands.sh to get a summary or look at the raw command line lists in build/logs.

  The Great Refactoring continues in sources/sections: broke up binutils-gcc.sh into individual binutils, gcc, and ccwrap stages, split off linux-headers and uClibc++ from uClibc.sh, and spun off toybox.sh from busybox.sh.

  Also broke kernel_cmdline() out of qemu_defaults() to make it easier to add non-qemu sources/targets configurations without reinventing the wheel, and fixed up dependencies for hw-target builds (which should work a lot more reliably now).

  Bumped up the number of $CPUS used by the build to 1.5x the actual number (as long as your host has at least 512 megs of ram per actual CPU). That should help keep larger servers busy.

  Some self-hosting fixes from Natanael Copa (making sure host-tools.sh builds on a uClibc host): added --disable-nls to the e2fsprogs build (what exactly is ./configure running all these tests for again?), and a squashfs patch (substituting a function uClibc hasn't got).

  Simplified system-image.sh to use killtree() instead of mysetsid. Simplified the cross-compiler/cross-static logic to set the $PATH to use the right one, rather than moving them around during the build. Updated smoketest-all.sh to show "NONE" for system images that didn't build. Added more support for SKIP_STRIP and CFLAGS=-g to the rest of the packages. Cleaned out the old USE_COLOR stuff which was superseded by set_titlebar.

November 7, 2009

Version 0.9.8 is out, based on hg commit 876. (If you want to see all the changes in this release, look at commits 810 through 876.)