Gentoo Package Management with Portage

June 9, 2004

This article was contributed by Ladislav Bodnar

Before Gentoo Linux emerged (pun intended) as a powerful force on the Linux distribution scene, software installation on most distributions was a subject of much controversy and frequent criticism. The RPM package manager was often at the receiving end of the blame due to its complex dependency structures; typing "rpm dependency hell" into Google's search engine returns over 700 entries. Debian's implementation of its own package management was often seen as technically superior to those found in RPM-based distributions, but many potential users were put off by Debian's other complexities and a lack of graphical installation and configuration utilities. And while Slackware's own package management does not suffer from dependency issues (simply because it doesn't care about them), many will argue that its pkgtool and other package utilities are rather too basic to qualify as a package management tool in the true sense of the word.

Enter the world of Gentoo Linux. With one short command, a user can install any application, without ever having to worry about libraries and other dependencies. The command will download the source code of the requested package, it will also download the source codes of all the dependent packages (if any), then it will configure, compile and install the package(s) without any further intervention on the user's part. Because the vast majority of packages are compiled directly from source code (with some exceptions, such as OpenOffice, Java, Opera, etc.), the user has complete control over package versions, optimizations, even optional dependencies, in some cases. In fact, the only drawbacks when compared to binary package management tools are longer installation times and fairly stringent hardware requirements - the pleasure of using Gentoo Linux rises exponentially with the power of the CPU at hand.

Portage. The main force behind Gentoo's convenient package management is Portage. Written in Python, Portage not only provides the all-important "emerge" command, it also groups all packages, or ebuilds in Gentoo speak, into a logical tree-like structure in /usr/portage/. Ebuilds can be unmasked, masked or hard masked, with unmasked ebuilds considered stable and well-tested, while masked ebuilds are stable, but not yet tested extensively. Hard masked ebuilds are usually reserved for alpha or beta packages. Only unmasked packages are installed by default, but this can be overridden either in /etc/make.conf or on the command line like this:

# ACCEPT_KEYWORDS="~x86" emerge packagename

The list of current masked packages is stored in /usr/portage/profiles/package.mask. To see the available versions of a package and their masked status in a nicely colored output, we can use the etcat command (part of the gentoolkit package):

# etcat -v mozilla

USE-flags. The USE-flags is a very clever concept, serving as a central place to configure some of the most basic aspects of the operating system and its behavior. As an example, if you prefer to have support for ALSA for all the relevant applications, you have to specify it as a USE-flag. Once done, these applications will be compiled with ALSA support included. If you don't specify it centrally in /etc/make.conf, you would have to compile the applications with the following command:

# USE="alsa" emerge xmms

The above example is still useful in case a user wants to override the global setting. There are over 200 USE-flags listed in /usr/portage/profiles/use.desc; they provide many "uses" that range from Java and SELinux support to specifying the default Japanese input server or enabling potentially offensive items in packages. The USE-flags also specify one of the CPU architectures - the currently available choices include x86, ppc, ppc64, sparc, alpha, mips, hppa, arm and amd64.

CFLAGS. Much has been said about CFLAGS on various forums, especially by those who never read the GCC manual. Probably the most important decision here is the use of march or mcpu when specifying the type of processor. The former will result in faster binaries at the expense of compatibility, while the latter will be produce somewhat slower binaries, but will cover a wider range of processors. As an example, specifying -mcpu=i586 will result in binaries optimized for i586, but they will also execute on i386; however specifying -march=i586 means that the binaries will only execute on i586, but not on i386 CPUs. Another common CFLAG is -O, which specifies a level of optimization ranging from -Os (optimized for small binaries) to -O1, -O2 or -O3 (optimized for speed). GCC provides over a hundred different CFLAGS.

Examples. Finally, let's have a look at some practical examples to illustrate the range of available options in Portage. The handful of the commands listed below are just a tip of the iceberg.

# emerge-webrsync

This is the same as emerge sync (it synchronizes the local portage tree with the latest available portage tree on official repositories), but useful for those users who are behind a firewall blocking the standard rsync port 873

# nice -n19 emerge -u world

This is the same as emerge -u world (it updates all installed packages to their latest available versions), but with a low priority so that the compilation process has a limited effect on the user's ability, in terms of processor resources, to perform other tasks.

# emerge -pv packagename

Lists packages that are dependent, but not yet installed, on packagename.

# emerge /mnt/cdrom/packagename*tbz2

Installs a binary package packagename from the mounted CD-ROM.

# emerge info

Lists all the Portage-specific variables, such as USE, CFLAGS, COMPILER, GENTOO_MIRRORS, etc.

If you are used to the rpm command, you can install Gentoo's epm to query packages. It takes many of the same arguments as rpm itself:

# epm -q packagename

Returns the installed version of packagename

# epm -qa

Returns all installed packages

# epm -qi packagename

Returns information about packagename

# epm -Va

Checks the integrity of all installed packages and reports conflicts

# epm -e packagename

Uninstalls packagename, same as emerge unmerge packagename

Learning about Portage is a journey in itself, a road paved with many pleasant surprises behind every turn. There is no doubt that Portage is one of the most powerful package management utilities available today.

Index entries for this article
GuestArticles	Bodnar, Ladislav

Gentoo Package Management with Portage

Posted Jun 10, 2004 2:20 UTC (Thu) by smoogen (subscriber, #97) [Link]

While Gentoo is better the larger the CPU.. it is also better the larger the network connection. I wouldnt try it on a 28.8 or slower modem connection...
Hopefully they will run a phone line capable of 56k out here in the boondocks sometime.

Gentoo Package Management with Portage

Posted Jun 10, 2004 3:11 UTC (Thu) by yodermk (guest, #3803) [Link]

>>> # ACCEPT_KEYWORDS="~x86" emerge packagename

Actually, that's not the recommended way of doing it. That will emerge the "beta" version once, but can confuse the system when upgrading. If you do an "emerge -u world", it will likely downgrade to the stable version. If you do an "emerge -U world", to force only upgrades and never downgrade, it could break your system by not reverting packages later found to be buggy. That is, however, how I did it, until I discovered this thread in the Gentoo forums:

http://forums.gentoo.org/viewtopic.php?t=163377

This is a must read for every Gentoo user. It starts getting good with robmoss2k's second post, about a fourth of a way down the page.

Portage is incredible. I'm nearly infinitely glad I switched from Red Hat about a year ago. :) I still use Fedora at work, but maybe I'll get them on Gentoo some day. :)

Gentoo Package Management with Portage

Posted Jun 10, 2004 3:22 UTC (Thu) by piman (guest, #8957) [Link] (4 responses)

> many potential users were put off by Debian's other complexities and a lack of graphical installation and configuration utilities.

And Gentoo has graphical installation and configuration utilities? (Hint: the official installation manual outlines manual command lines to run to do the install.) Anyone put off by Debian woody's installation process should be scared to death of Gentoo's.

Gentoo Package Management with Portage

Posted Jun 10, 2004 3:38 UTC (Thu) by yodermk (guest, #3803) [Link] (3 responses)

Actually, I found Gentoo's installation and configuration documentation truly excellent, almost something by which other distributions and open source packages should be judged. It does take some editing files, but the docs should make it easy for just about anyone, if they set their mind to it a bit.

Another reason to be put off by Debian is the ancientness of the packages. Sure, there's "testing" and "unstable," but with Gentoo you can decide to run "testing" or "unstable" programs on a package by package basis. In Debian, to get one unstable program, you have to pull in the unstable glibc, which is scary on a production server. Since Gentoo compiles on the fly, that isn't a problem.

Sorry to sound like a fanatic, but Gentoo really *is* the best thing since sliced bread. :)

Gentoo Package Management with Portage

Posted Jun 10, 2004 7:00 UTC (Thu) by piman (guest, #8957) [Link] (2 responses)

Gentoo's documentation may be excellent, but so is Debian's. If a user is going to be scared off by Debian's install, they will also be scared off by Gentoo's. I just don't see the point of mentioning Debian's installation process in an article about package management tools.

Gentoo Package Management with Portage

Posted Jun 11, 2004 1:19 UTC (Fri) by yodermk (guest, #3803) [Link] (1 responses)

I dunno -- I've done some with Debian also, and frankly, I thought Gentoo was easier.

Gentoo Package Management with Portage

Posted Jun 21, 2004 21:58 UTC (Mon) by k8to (guest, #15413) [Link]

I think this was true, once upon a time.
Debian has too many paths to installation (and everything else) to make any of them well trodden enough to be absolutely streamlined. However, the new installer for sarge really makes things simple.

Me? I use debootstrap for the one command install.

Gentoo Package Management with Portage

Posted Jun 10, 2004 4:54 UTC (Thu) by modernjazz (guest, #4185) [Link] (2 responses)

I tend to run Debian systems installed through Knoppix, and one thing has
always bothered me: I'd like to be able to get security updates without
also updating all the other packages on my system (if it ain't broke...).
One can get security updates by following either "stable" or "unstable."
Unfortunately, "stable" is quite old and doesn't update many of the
packages found on Knoppix; and in "unstable" the packages seem
to be replaced rather often (for many reasons besides security), so
staying current involves a lot of "package churn."

Does Portage provide a way to run current software and apply security
updates without the amount of package churn I see in Debian unstable? Or
perhaps there's something I'm missing in how to configure apt?

Gentoo Package Management with Portage

Posted Jun 10, 2004 6:24 UTC (Thu) by piman (guest, #8957) [Link]

You can backport Debian packages relatively easily (as easy as you can recompile Gentoo packages for older libraries) by doing 'apt-get -b source <source package name>'. You can pin, specify releases, whatever, the same way you do for binary pacakges; APT will download and compile the source package, and give you a .deb. If the source itself isn't dependant on a newer library version, it will work with your older libraries.

The 'apt-src' and 'apt-build' tools also give you a way to manage Debian as a set of source packages rather than binary ones.

Security Updateds Only?

Posted Jun 10, 2004 7:47 UTC (Thu) by Felix.Braun (guest, #3032) [Link]

Gentoo are working on an update command that will only pull in security related updates called glsa-check. The documentation states that there are currently some remaining shortcomings to be ironed out. I can't say for myself as I haven't used this yet.

Ladislav Bodnar's reporting

Posted Jun 10, 2004 5:54 UTC (Thu) by popinet (guest, #5248) [Link] (11 responses)

I am sorry to say to that I do not consider Ladislav Bodnar reporting as matching the overall quality of LWN reporting. As one other reader put it he seems to be rather too excitable about the subject at hand. This article about portage is no exception. It may be a matter of personal taste (nothing personal Ladislav) but I would rather have a calmer (more objective?) discussion about the pros and cons of portage (or any other package) rather than "enthusiastic marketspeak").

Ladislav Bodnar's reporting

Posted Jun 10, 2004 10:45 UTC (Thu) by copsewood (subscriber, #199) [Link]

I enjoy Ladislav's articles, and enjoyed this one. I found this one a very consise and useful overview of managing Gentoo. I'm not yet ready to use Gentoo for mainstream work but would like to find the time to play around with it. I agree that this article doesn't compare against other types of installation, but presumably those who don't want to install most things from source which they compile themselves will be able to figure out that they don't want to use Gentoo from other more comparitive and general purpose distro-related articles.

Ladislav Bodnar's reporting

Posted Jun 10, 2004 16:52 UTC (Thu) by ballombe (subscriber, #9523) [Link] (3 responses)

I agree. Not citing apt-get in an article about packages management seems odd.
A comparison between apt-get and portage in term of behavior would have been
welcome.

Doing a mental s/Gentoo Linux/apt-get/ in the quote below is enlightening:

"Enter the world of Gentoo Linux. With one short command, a user can install
any application, without ever having to worry about libraries and other
dependencies."

Ladislav Bodnar's reporting

Posted Jun 11, 2004 14:19 UTC (Fri) by ekj (guest, #1524) [Link] (2 responses)

Actually, the entire thing is years out of date.

These days, every serious linux-distro provides package-management that deals with dependencies.

It is true that people *used*to* complain about this years ago, but these days the only reason you'll see more incompatibilities and messed dependencies for *rpm than for *deb is that there's a lot of different distros using rpm, and basically only debian using deb.

Thus you risk getting stupidity when trying for example to install a redhat-rpm on mandrake. Perhaps redhat uses foo-1.2.70 and the redhat-package you want to install depends on it, while mandrake has upgraded to foo-1.4.3, and kept on the older one as foo12.70 for compatibility-reasons.

End result; the package will claim that you ain't got foo >= 1.2.70, even though you do. This ain't an rpm-problem but a problem caused by different distros giving different names to the same stuff.

"urpmi gimp" does exactly the same thing as "apt-get install gimp" and urpmi --auto-select does the same as whatever command the debianers use to update all packages to the latest version.

Ladislav Bodnar's reporting

Posted Jun 11, 2004 18:30 UTC (Fri) by piman (guest, #8957) [Link] (1 responses)

> these days the only reason you'll see more incompatibilities and messed dependencies for *rpm than for *deb is that there's a lot of different distros using rpm, and basically only debian using deb.

There's a lot of distributions using .deb, actually; the difference is that Debian has an extensive packaging policy, which anyone can read and use. So all the third party distributions have packages that more or less follow that policy. The other distributors also usually work with Debian to the extent that their goals allow. Policy is the real strength of Debian's packaging system, and the reason that you don't find many incompatible third party .debs.

> "urpmi gimp" does exactly the same thing as "apt-get install gimp" and urpmi --auto-select does the same as whatever command the debianers use to update all packages to the latest version.

Last time I checked, RPM (and URPMI) had no equivalents of APT pinning (manage specific sets of packages from other sources), large-scale downgrading, deborphan and debfoster (automatically remove unneeded packages), auto-apt (automatically install packages when something on the system requests a file from a package), debhelper, (a consistent way to create packages), and a dozen other tools that make up the Debian packaging system.

It's true that most other distributions have caught up to the level of sophistication that Debian's packaging tools were at -- 4 years ago. Installing and upgrading packages is the *bare minimum* a package manager should be able to do.

Deb vs. RPM

Posted Jun 14, 2004 16:41 UTC (Mon) by EricBackus (guest, #2816) [Link]

And yet, deb (or rather, apt) still doesn't automatically handle
cryptographically signed packages, right? So you can auto-update,
downgrade, whatever, but you don't *really* know what you have when you're
done.

Ladislav Bodnar's reporting

Posted Jun 11, 2004 13:25 UTC (Fri) by ladislav (guest, #247) [Link] (4 responses)

Thank you for being honest. Like any author, I'd much rather see praises and compliments from my readers, but of course that could only happen in an ideal world. The truth is that I actually agree with you.

I have been researching Linux distributions for over 3 years. The result of this research is that I know something about every distribution, but I am not an expert on any of them. Unfortunately for me, the readership of LWN is full of experts; there are readers who have been using Red Hat (and Fedora) for years, so there is little they don't know about it. When I have to write an article about Fedora, I need to boot into it, spend some time investigating the new release, research the mailing lists and forums, even ask developers or people who are more knowledgeable on the subject. But even if I spend all day in Fedora, there is no way I can match the expertise of those who have been using nothing but Fedora since its first release.

And that's true about any other distribution. I run Debian Sid as my production system and have about 25 Linux and BSD partitions on my two hard disks with all major and interesting distributions on them. I know a lot about them on surface, but I don't have any deep knowledge of any of them, except perhaps for Debian.

Which leads to the following question: am I writing for the right audience? Since it would appear that the vast majority of LWN readers are seasoned experts on whatever Linux distribution they have settled on, there is no way I can teach you anything new. Perhaps my articles would be more suitable for users who are relatively new to Linux or for those who are still deciding on a distribution. Or perhaps I should write about more general distribution topics without attempting to go too deeply into technical details. I don't know.

In a word, I am considering quitting as a LWN contributor. I do enjoy writing and I do enjoy spending extra time in different distributions while researching their technical aspects. But writing is also stressful. It basically kills my every Wednesday, a day which is normally very busy on my own web site. I am not a native English speaker either, so writing and revising (and revising again...) takes me a lot longer than it would take most native speakers.

But before I quit, I'd love to hear your opinions. If most of you feel that my articles do not match the high standards of LWN, then I'll quit. But it's also possible that, except for a few vocal moans, the majority of the readers are quite happy with my stories, in which case I'll continue contributing to LWN. If you can think of any improvements, please mention them too. I'll make a decision after I read your feedback (if any).

Ladislav Bodnar's reporting

Posted Jun 11, 2004 18:46 UTC (Fri) by piman (guest, #8957) [Link]

Perhaps you should try different kinds of articles; out of those 25 distributions (and many more you've probably tried), there's got to be some that no one has heard of. For example, I enjoyed your Gobo Linux article; although I don't agree with the goals of the design, the article itself was informative and interesting. YA package manager isn't interesting, but an attempt to replace the whole Unix filesystem is.

I'd also be interested in direct comparisons; I have absolutely zero idea how yum works or what it can do, besides some vague notion that it solves some of the same problems as APT.

Ladislav Bodnar's reporting

Posted Jun 13, 2004 23:21 UTC (Sun) by sweikart (guest, #4276) [Link]

I agree that "even the experts" will probably learn about minor distros on which you write reports. In addition...

Even if many LWN readers are more expert then you on a couple of distros, most of us are probably not expert on all the major distros. So, most of us (including me) will find most of your articles informative, which is very useful (to me).

-scott

Ladislav Bodnar's reporting

Posted Jun 14, 2004 17:03 UTC (Mon) by EricBackus (guest, #2816) [Link]

You had the bad luck to pick a topic--package managers--that generates
lots of passionate opinions (look at the number of responses your article
generated!). Because of that, articles about package managers have been
written before and many people already know a fair amount about them.

While I know redhat and suse fairly well, all I know about gentoo is from
reading articles like yours and the comments that such an article
generates. I know enough to know that I don't want to try gentoo myself,
but it still seems like a good idea for me to know more about it.

So, on the whole I liked your article and would prefer that you keep
writing. I know the LWN experts are intimidating, but nobody is an expert
at everything.

As for suggestions, it would have been helpful to show the equivalent rpm
or apt command for each of the portage commands you gave, or talk about
what specifically (if anything) makes portage different than those.
That's potentially a lot of work, but would have been useful to me.

Ladislav Bodnar's reporting

Posted Jun 18, 2004 22:57 UTC (Fri) by roelofs (guest, #2599) [Link]

But before I quit, I'd love to hear your opinions. If most of you feel that my articles do not match the high standards of LWN, then I'll quit.

While it's certainly true that the tone of your articles doesn't match that of Jon's, and while I very much enjoy his style of writing, I don't find yours deficient in any way; it's just different. For the most part I've truly enjoyed and appreciated the content of your articles. I may be a dyed-in-the-wool, 10-years+ Slackware hacker myself, but I also like to keep abreast of the latest developments in other distros, both for myself and for recommending distros to friends and relatives. It's because of your coverage that I've gone ahead and experimented with KNOPPIX-to-disk, several of the specialized "live" CD distros (including a do-it-yourself variant that was an absolute godsend), and a couple of the "old-hardware/limited-resource" distros. I'll also probably give Gentoo a go on another system in the not-too-distant future, partly because of this very article.

In short, I encourage you to stay...but having been on the receiving end of a lot of ungrateful whining myself, I certainly wouldn't fault you for leaving. It's basic human nature that people are never completely satisfied with what they've got--it's always got to be a little bit more, a little bit better--and that can be difficult to take when they want it from you. :-) But I suspect there are a lot of us who truly appreciate your efforts, not least because we don't have anywhere near enough time to do this kind of review and experimentation ourselves.

Greg

Ladislav Bodnar's reporting

Posted Jun 11, 2004 23:40 UTC (Fri) by pimlott (guest, #1535) [Link]

I'm afraid I agree. This article should go somewhere on the gentoo site. I expect LWN articles to make connections I won't find anywhere else and take a provocative but convincing perspective--and be artfully written to boot. This is an exceptionally high standard, and Ladislav's articles generally don't measure up.

Gentoo Package Management with Portage

Posted Jun 10, 2004 6:55 UTC (Thu) by piman (guest, #8957) [Link]

It's disappointing to see -O3 mentioned without the caveat that much of the time, it can make code take longer by blowing icache, and register renaming makes debugging a pain.

It's true that much is said about CFLAGS; and the safest thing to do by far is to turn on -O2 and leave it at that. Knowing what the "correct" optimizations are for a piece of speed-critical code involves more work and knowledge about how machine code is generated than most users have, and arch-specific optimizations tend to trigger complex and subtle compiler bugs, that most users and developers are not prepared to debug.

Gentoo Package Management with Portage

Posted Jun 10, 2004 15:39 UTC (Thu) by raytd (guest, #4823) [Link]

Slackware's own package management does not suffer from dependency issues...

I wouldn't say that exactly. pkgtool doesn't suffer dependency nightmares, but it does sometimes force the administrator to suffer from them.

No reverse dependencies

Posted Jun 10, 2004 19:00 UTC (Thu) by shredwheat (guest, #4188) [Link] (1 responses)

I consider emerge's lack of reverse dependency handling a major problem. This makes it near impossible to safely remove or downgrade any package. For example, once you decide to start using the "~" testing versions of packages there is no going back.

This makes using gentoo like a one way road. It's a very nice and polished one way road, but once it takes you somewhere you don't like, your only option is to keep pushing forward and hope you eventually get past it.

I'm surprised an article about package management on Gentoo fails to mention this. But this entire article seems more of a fluff piece than an objective review. I was turned off after "Gentoo's package management is better than Debian's since people don't like the Debian installer." Yeah, ok, and Gentoo's installer is what?

No reverse dependencies

Posted Jun 13, 2004 9:07 UTC (Sun) by evgeny (subscriber, #774) [Link]

> I consider emerge's lack of reverse dependency handling [...]

Not exactly. qpkg -I -q <package name> will give you this info (though not recursively).

> For example, once you decide to start using the "~" testing versions of packages there is no going back.

Why? I once switched from ~amd64 to amd64 and can't remember any major issue. Having said that, ask yourself what's the reason one chooses an unstable variant of a distribution. With Debian, many people opt for "testing" or "unstable" because the "stable" brunch is hopelessly outdated for about any purpose, using backports is not widely known (and is quite restricted in number of packages available), and mixing of stable + unstable is a real hell for any binary distro. Contrary to that, with Gentoo it's extremely easy to have such a mix. Further, difference between stable and unstable in Gentoo is an order of magnitude less than ~100% in any binary distro. As a result, it's pretty common to have only a few packages specially marked via /etc/gentoo/packages.*. These are things one uses in daily work and ultimately needs a specific or most-up-to-date version. If a newer version of such a package is broken, it's a matter of downgrading only this specific package.

My personal experience with Debian/unstable vs Gentoo: the latter is more stable and has given me _much_ less pain with upgrading/downgrading/dependency resolving issues. And that's what counters, IMHO.

Gentoo Package Management with Portage

Posted Jun 17, 2004 9:21 UTC (Thu) by job (guest, #670) [Link] (1 responses)

I would hold LWN to higher standards. As people have pointed out, this
article containts newbie mistakes. You could have asked ONE more
experienced Gentoo Linux user to proofread it before publishing.

An article about Portage should basically cover the concept of USE flags,
the stable/unstable flag, how to clear old versions, using /etc/portage,
the files left over in tmp work and distfiles. It should then dwelve into
more advanced concepts such as how sandboxing works, what to do when it
fails, the ebuild syntax etc. I can say this and I haven't even used the
thing for a year!

When comparing with other distros I'd be more interested in how they work
in practice, how many tools there is to care about (Gentoo has a couple
too many), how upgrading works (Gentoo isn't really there yet), how
current it is kept, and so on. Your article points to Portage having
superior dependencies which simply isn't true.

Gentoo Package Management with Portage

Posted Aug 12, 2004 18:31 UTC (Thu) by mrness (guest, #8271) [Link]

I've been a Gentoo user for more than a year and I definitely think that "Gentoo is there" where it belongs. Never had problems with upgrades or downgrades (where was the case). Well, maybe excepting my human mistakes. In any case, I don't blame Gentoo tools for that; the fault was mine.

And yes, Gentoo HAS superior dependecies. Name a single other package manager in which I could modify dependencies of a package by changing a variable (USE)! Only with source based distribution such features are possible. You don't want samba to be involved in other's packages dependency list? Simply add -samba to USE!