Editing FreeBSD Reference (section)

= Ports =

This third lesson deals with the FreeBSD ports tree, which is software packaging/installation mechanism that is used in FreeBSD.  You may remember that, in the few times we have installed FreeBSD over the phone, one of the components we always install is "Ports" - this is the ports tree.

In general, across all unix variations, the method for installing new software is always about the same - you download a source tarball and unzip/untar it, run ./configure, make, make install.  Sometimes it is a bit different, but that's the general procedure.

As you know, there are shortcuts in every operating system to compiling and building a package.  For instance, in solaris, you can get pre-packaged binaries that are already compiled and install them with the `pkgadd` program.  Or, even more popular, in Linux you can use the `rpm` command to install RPM files that are also prepackaged, precompiled binaries for linux.  Another popular method is the Debian method which uses .deb files, I believe.  The important thing about all of these shortcut methods is that they are a manner of packaging up _precompiled_ binaries - you don't actually compile them when you install them.  pkgadd, rpm, etc., are all just ways of copying and placing files (and chmodding them, etc.)

The ports tree is different.

First, let's take a look at the ports tree and see why it is called a "tree".  Run this command:

 cd /usr/ports

then do an `ls`.  You will see a fairly large list of directories, and each directory is obviously a category of software (sysutil, security, ukranian, cad, ftp).  Now run this command:

 cd /usr/ports/sysutils

Now run `ls` again - you will see that in the sysutils directory there are all sorts of ... system utilities.  Who knows what they all are or what they all do, however you will see some familiar ones ... rdate, logrotate, httplog, etc.  Now run this command:

 cd /usr/ports/sysutils/rdate

Now run `ls` again.  This is the ports directory for the rdate application.  As you can see there is a distinfo, pkg-descr and pkg-comment file - if you read those they will tell you about this distribution, what rdate is and what it does.  There is also a "work" directory, and most importantly a Makefile.  But, please note, there is no package file and no source code - the package is not here - so what is all this for and what does it do ?

Run these commands:
(make sure name resolution works on your system first)

 cd /usr/ports/sysutils/rdate
 make install clean

Now, watch what happens - suddenly you are fetching files over ftp ... those files are being untarred, and compilation is occurring ... rdate is rather small, so in a short time it should be all finished.  Now run the command `rehash` to get your shell to take note of the new `rdate` command that is now in your path (alternatively you could log out and log back in).

Now run <tt>rdate -s time.nist.gov</tt>

rdate has now been installed.

So, what happened ?  Well, open up the Makefile for rdate in a text editor
and note the line that says:

<pre>MASTER_SITES=   http://www3.cons.org/freebsd-distfiles/ \
                ${MASTER_SITE_LOCAL}
MASTER_SITE_SUBDIR=     cracauer</pre>

So, when you ran <tt>make install clean</tt>, the Makefile had the information as to where to go to _get_ the source tarball.  It downloaded the source into /usr/ports/distfiles (run <tt>ls /usr/ports/distfiles</tt> and you'll see it there) then it untarred it, and configured and installed it, just as if you were compiling the package by hand.  The clean target for make causes the work dir to be removed after compilation so that no extra disk space is wasted.

So, the big difference between the FreeBSD ports tree and the other methods on other unixes (rmp, pkgadd, deb) is that you don't have a precompiled binary - you install from source, even though you are using the ports tree.

The other big difference is that it is arranged as a tree, as you have seen.  When you want to install `wget`, you:

 cd /usr/ports/ftp/wget
 make install clean

and when you want to install BitchX you:

 cd /usr/ports/irc/bitchx
 make install clean

And every time the source tarballs will end up in /usr/ports/distfiles

It should be noted that after you have successfully installed a port, you can just delete the tarballs out of /usr/ports/distfiles if you want to save the space....or you can leave them in case you ever install the port again, since they are already there, FreeBSD won't bother downloading them again.

So that's the primary explanation.  Most people that know FreeBSD know all of the above (although some people don't know to run `rehash` after installing to get the new binaries into your path - they just log out and log back in).  It is interesting to note that regardless of ones own preference concerning which OS to use, almost _everyone_ (even hardcore linux zealots) concede that the FreeBSD ports tree is the nicest, easiest, and most configurable way to do package management.  It is considered one of the main advantages of using FreeBSD, from a user standpoint.

So let's move on to some advanced topics concerning the ports tree.

First, <tt>make install clean</tt> can fail.  Rdate was a very simple program to install, but sometimes you install something very complicated, and it depends on other packages, and those other packages depend on even other packages.  Now the nice thing is, the ports tree manages all of this – if you install a package with a lot of dependencies, it installs all of them automatically - and installs all of their dependencies automatically as well.  All you have to do is go to the single port you want and run <tt>make install clean</tt> and it does them all in the right order, etc.  It's very slick.

But, sometimes a port (like rdate) has only one master site listed in the Makefile (a lot of ports have many sites listed) and if that site is down, it can't download the source tarball, and obviously it fails.  Another common problem is that you have a newly installed FreeBSD machine and you forget to populate /etc/resolv.conf so you have no name lookups – which means when you <tt>make install</tt> and it tries to connect to ftp.example.com it fails.  Further, you could have an old ports tree from an old installation that asks for some_software-1.2.3.tar.gz, and the ftp site no longer has that old file there anymore - they only have 2.3.4...so again it will fail.

Regardless of where it fails though (maybe the main package failed, or maybe a dependency of a dependency of a dependency fails) you clean things up simply by going to the ports directory that you initially ran <tt>make install</tt> in and just run <tt>make clean</tt>.  Whatever error messages you saw when it bombed out will tell you what port it was on that it bombed out on (because again, it might bomb out not on the port you are installing, but on some dependency of the port you installed) and it will give a clue as to why it bombed out.  So use that info, fix the problems, run `make clean` and then run `make install` again.

Second, you can populate /usr/ports/distfiles by hand.  So let's say you <tt>make install clean</tt> and it bombs out because it can't download a file (maybe the ftp site is down).  If you can find that exact same file somewhere else on the net, you can download it into /usr/ports/distfiles and run <tt>make install</tt> again - it will find it and just use it.

Third, installing a port over itself - let's say you install a port, and then either want to reinstall it later, or reinstall it with a newer ports tree (more on that later) all you have to do is go into the ports dir for that app (for instance /usr/ports/sysutils/rdate) and run <tt>make install</tt> again.  Although there may be exceptions to this, and if there are major differences in version A of an app and version B, maybe it won't always work.  But in almost all cases, you can just go back to the port directory for that app and run <tt>make install</tt> again.

Fourth, if you want to look at what ports are installed on a particular FreeBSD system, just run the command <tt>pkg_info</tt> and you will see a list of all ports installed on the system.  However, this info is extracted from the /var/db/pkg, so if you delete /var/db/pkg you won't get any info.

Fifth, you can change the way that port compiles and/or installs by editing the Makefile in the directory of that port.  This is fairly rare that you need to do this, and in most cases, like you saw with `rdate` there is not even much to the Makefile itself.  However many times there are arguments you can change in the Makefile to alter the way that the port compiles and eventually installs.

However, more commonly, you alter the way that a port installs by adding additional command line arguments to the normal <tt>make install clean</tt>.  For instance:

Edit the Makefile in /usr/ports/net/cvsup and note this line:

 .ifdef WITHOUT_X11

This tells us that, if we wanted to, we could install this port with the command line:

 make WITHOUT_X11 install

Why would we do this ?  Well, cvsup is a very common utility that allows you to sync or install files and filesystems from servers called cvs repositories (cvs is a software version control system, and it is the very popular, de facto standard for all large open source software projects (including the linux kernel and FreeBSD/NetBSD/OpenBSD)).  However, for some dumb reason when you go to /usr/ports/net/cvsup and run <tt>make install</tt> the port also installs all these silly X utils for running cvsup ... and of course, all those X utils depend on a lot of X binaries and libraries.  Many people are shocked when they try to install the simple cvsup utility and watch the port install 100 megs of X dependencies for some lame helper apps they will never even use.

Thus, most folks install cvsup with <tt>make WITHOUT_X11 install clean</tt>.  This is just a small example of why you might add additional arguments on the <tt>make install clean</tt> command line to alter the way a port either compiles or installs.

Additionally, if you know you’ll never have X11 installed, you can put

 WITHOUT_X11=yes

into /etc/make.conf and that will be defined for any port that you build.

Sixth, you can actually patch a software package from within the ports tree.  As you remember, we used the `patch` command to patch the source code for the FreeBSD kernel.  Usually, as you saw, we always have a "patch file" with the "unified diff" syntax that we run the patch command against.  Well, in the world of compiling your own packages from source tarballs, it is very common to patch the source for a software package just like we patched the code for the FreeBSD kernel.

Now, since the ports tree actually downloads the real source tarballs and does an actual compilation of the source code (unlike rpm or pkgadd which just copies in precompiled binaries) it stands to reason that if we need to patch a packages source, we could do it even though we are installing from the ports tree ... and we can.  However, this is somewhat rare.  The reason it is rare is that if you are meticulos enough to be patching a particular version source for a package, you are probably just installing it by hand anyway.  I have only ever patched a port once in my life.  But just for your information, you just take the patch file and place it into:

 /usr/ports/(category)/(port)/files

and just run <tt>make install clean</tt> as usual.  The port will see that there is a patch file in the files subdirectory and attempt to apply it automatically.  Easy.

Now, the last advanced topic to understand about the ports tree is _updating_ the ports tree.  Let's say you install FreeBSD 4.4-RELEASE and you don't ever upgrade it, but everything is ok and you don't feel the need to upgrade.  However, a new version of XYZ app comes out and for some reason you want to just install it from the ports tree .... but your ports tree that got installed with 4.4-RELEASE is an older one, and only knows to download and compile the older version of XYZ app.  What to do ?

Well, since you do not want to upgrade or reinstall your OS, what you need to do is update your ports tree.  I am not going to provide the step by step for that here, since I have a URL for the process that works very well, but here are some pointers for the process:

1. It involves using the cvsup command we mentioned earlier.  You should install this, and use the WITHOUT_X11 switch most likely.

2. you can update only certain categories of the ports tree – for instance, there is little reason to update the entire ports tree and get all the "ukranian" and "math" and "russian" ports - you can instruct cvsup to just update a particular category like: /usr/ports/sysutils

3. If your version of FreeBSD is old (like 3.1) and you update to a new ports tree from a current version like 4.7, obviosuly some apps you install from the new 4.7 might be confused.  However, the beauty of the ports tree is that that should happen only very rarely - remember that when you install the port a full configure and make (compilation) occurs, so the compilation process should compile for the version you are running, and all should be well.

The instructions for updating your ports tree are here:

http://www.freebsddiary.org/ports.php

Please read it and pick some ports category you will never use like /usr/ports/chinese and update that with cvsup.

Also, if you are ever wondering what the vintage of your particular ports tree is, you can:

 cat /var/db/port.mkversion
 20020814

and see the date.  If port.mkversion is not existent or is an older date than some ports expect it to be, they will not install.  You can either solve the problem that caused it, or you can just edit that file and put in some later date and things will work again.