A couple of months ago, AI and I ventured down a rabbit hole
investigating how Perl distributions are located and retrieved from
CPAN and private repositories. That adventure resulted in a two-part
series, Reflecting on
Mirrors. In
a somewhat related side tunnel I managed to crack Apache2::Request’s
enigmatic build and installation process when using cpm, eventually
resulting in patches to both
libapreq2 and
cpm.
Emerging from the tunnel and basking in the sunlight of discoveries was, apparently, a short-lived reprieve from the mysteries and intricacies of the Perl toolchain.
In my never-ending unintentional discovery of oddities, I bumped into
yet another one this past week. The situation revolved around trying
to understand why .packlist was losing entries from previous
installations.
First, a little history.
Doing a bit of research on the old issues list
for ExtUtils::MakeMaker I came across an interesting
discussion
titled “packlists aren’t pure.” Strap in, turn on your flashlight,
we’re going down the tunnel.
The author of the issue questioned MakeMaker’s practice of reading an existing
.packlist and merging its contents with the inventory of files being
installed in a new version.
He posits, “Why not simply write a new packlist every time, as Module::Build does?”
It’s a reasonable question, and one the designers of
ExtUtils::MakeMaker presumably considered. While the implementation
doesn’t explain their reasoning, the decision to merge appears
deliberate.
Why would you want to preserve the list of files in every distribution? The immediate answer is revealed when you ask: How can I remove all of the files installed by a distribution?
Consider a distribution that evolves through three releases:
| Release | Files included |
|---|---|
| 1.0 | foo.xml, bar.xml |
| 1.1 | bar.xml |
| 1.2 | baz.xml |
If we merge successive installation inventories, the resulting packlist contains all three files.
If we replace the inventory on each installation, the packlist
contains only baz.xml.
Which behavior is correct?
Well, that depends on what you believe a packlist is supposed to represent.
Installing a Perl distribution is a relatively straightforward affair. The distribution declares its dependencies, an installer resolves them, and the files end up somewhere you specified-hopefully somewhere Perl can find them.
Uninstalling that distribution is another matter entirely.
For starters, Perl doesn’t maintain the kind of centralized package database found in RPM or Debian-based systems. A typical Perl application may depend on modules supplied by the operating system, modules installed from CPAN, and modules vendored locally with the application.
Perl’s @INC determines where modules are found, but it tells us
nothing about who installed them or who might still need them.
Then there’s the question of file ownership. Suppose distribution A
installs my favorite script, foo.pl, and distribution B subsequently
installs a script with the same name into the same directory. If we
uninstall A, should foo.pl disappear?
And what happens if B depends on a module supplied by A?
These are legitimate concerns that have led to a
rather curious solution. Using ExtUtils::MakeMaker:
…you can generate a Makefile and actually type:
make uninstall
But you may be surprised to discover that nothing actually gets uninstalled.
MakeMaker deliberately invokes ExtUtils::Install::uninstall() in
dry-run mode, warning that uninstallation is unsafe and
deprecated. Huh? And cpanm lists the --uninstall option as
experimental.
The underlying ExtUtils::Install function can in fact remove the
files. MakeMaker simply chooses not to let its generated target
(uninstall) do so. I suppose that is a reasonable approach if your
goal is to make sure your footgun doesn’t fire.
And there’s another oddity: to invoke make uninstall, you need a
generated Makefile. That means retaining the build directory or
obtaining the distribution and generating its Makefile again, just to
attempt to uninstall something that’s already installed!
And as I said, cpanm --uninstall provides an alternative that works
directly from the installed distribution’s packlist, although the
command is mysteriously labeled experimental as if we’re waiting on
the results for something(?) to decide if it stays or if it goes.
I find both of these approaches a bit paternalistic. Dependency and ownership risks are reasons to warn users about uninstallation, not necessarily reasons to call it experimental or prevent it from actually working.
But perhaps there’s another reason cpanm labeled the feature
experimental. What if the inventory used to uninstall a distribution
isn’t reliable in the first place? Spoiler alert…it isn’t.
That brings us back to the humble .packlist.
If we’re going to remove everything a distribution installed, we first need to know what it installed. Not just today, but throughout its installation history.
And that is why the question of merging versus replacing packlists matters.
The argument in #173 was largely motivated by the needs of package maintainers. When constructing an RPM or Debian package, the packager wants an accurate inventory of the files belonging to that particular release.
Files from earlier releases don’t belong in that inventory.
Fair enough.
But installing a Perl distribution isn’t quite the same thing as installing an RPM or Debian package.
Operating-system package managers maintain centralized databases of installed packages, file ownership, versions, dependencies, and removal operations. They have considerable machinery dedicated to managing a package’s lifecycle.
Perl distributions don’t have an equivalent centralized ownership database.
There’s a fundamental difference between operating-system package
managers like RPM and dpkg and language-specific installers like
cpanm, cpm, or Python’s pip.
RPM and Debian packages are primarily intended to install and manage software within an operating system. Yes, alternative installation roots and other arrangements are possible, but that’s not how most application developers use them, assuming they use them at all. These tools generally operate within a centrally managed environment, frequently requiring root privileges to install or remove system packages.
Perl and Python inhabit something closer to the Wild West. Developers routinely install modules into their home directories, application-specific directories, virtual environments, or vendored dependency trees. New little towns spring up everywhere, each with its own collection of packages, and none necessarily aware of the others.
Perl compounds the situation by maintaining separate directories for
core, vendor, and site modules. Add local::lib and
application-vendored modules, and it’s not unusual for a single
application to resolve dependencies from three or four different
installation trees.
Perl knows where to look through @INC, but it doesn’t maintain a
centralized registry of who installed what, where, or who still
depends on it.
That makes the humble .packlist considerably more important than it
might first appear. For an independently managed Perl installation, it
may be the only enduring record of the files a distribution has
installed.
Unlike an RPM or Debian package, a Perl distribution cannot assume that some other package manager is keeping track of the files it leaves behind.
And that’s where replacing a cumulative packlist becomes problematic.
Suppose version 1.0 installed foo.xml, but version 1.1 dropped
it. If installation does not remove obsolete files, foo.xml is still
sitting on disk.
If the new installation also removes foo.xml from the packlist,
we’ve now lost track of a file we installed.
A subsequent uninstall cannot remove a file it no longer knows exists.
The two requirements are different:
Trying to make one inventory satisfy both requirements is where the trouble begins.
One could imagine versioned packlists:
.packlist.3.9.2
.packlist.3.9.3
.packlist.3.9.4
Each would provide an accurate inventory of a particular release. A complete uninstall could examine the accumulated inventories and remove the union of files installed by the distribution.
That would provide both a clean release inventory and a historical record.
Of course, this is an architectural change. Existing Perl tooling
expects a conventional .packlist, and introducing versioned
inventories would raise questions about backward compatibility, file
ownership, and uninstallation semantics.
Before anyone thinks I’m proposing a redesign of MakeMaker’s installation machinery to solve a potential problem, let me assure you I’m not!
For now, its existing cumulative packlist behavior seems a reasonable compromise.
Assuming it actually worked.
Let’s look at a real-life case where we can see if
ExtUtils::MakeMaker actually does
what it thinks it does.
Consider what happend when I restructured a CPAN distribution, moving functionality into more narrowly focused distributions. As part of that work, files were relocated or removed from the new release.
That raised an obvious question: what happens to those files in the older releases when we install the new releases of the now split package?
We don’t particularly want installations silently deleting files from earlier releases during installation. Installation and uninstallation should be distinct operations, with the latter deliberately initiated by someone who knows what they’re removing.
And…ExtUtils::MakeMaker’s implementation of make uninstall
which uses .packlist looks like it might help to uninstall
files…but runs in dry mode!
As we discovered earlier, MakeMaker’s make uninstall doesn’t actually uninstall anything. Fortunately, ExtUtils::Install provides the underlying operation directly:
use ExtUtils::Install;
ExtUtils::Install::uninstall(
'/path/to/auto/Bedrock/Web/.packlist',
1, # verbose
1, # dry run - change me to 0 to do the deed
);
So does it actually work? Well, yes and no. It works if you are
installing things into the system directory, but not when you use
INSTALL_BASE. Let me explain.
While examining the packlist for my newly split distribution I noticed that files removed from this new release were disappearing from the inventory after reinstalling the distribution into my local environment.
The files were no longer recorded in .packlist as they should have
been, even though they hadn’t been uninstalled.
Install the original version of a distribution containing some file and inspect its packlist:
grep bedrock\.pl /home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Bedrock/Web/.packlist
/home/rlauer/bin/bedrock.pl
Next, remove the file from the distribution, rebuild it, and reinstall it.
grep bedrock\.pl /home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Bedrock/Web/.packlist
Gone. Vanished. Poof! Hey! Where did it go?
Not necessarily from the filesystem, only from .packlist. And
bedrock.pl is now forever orphaned like a lost rabbit.
command -v bedrock.pl
/home/rlauer/bin/bedrock.pl
And that was precisely the behavior MakeMaker’s merging mechanism was supposed to prevent.
Could it be the installer itself? My first suspicion fell on cpm
since I use that as my installer these days. I knew that it used its
own installation machinery by default.
Perhaps it was simply replacing the packlist?
So I tried again with --use-install-command, directing cpm to execute
the distribution’s installation command.
Same result.
So I repeated the experiment using cpanm.
Same result.
At that point, either both installers were making the same mistake, or the problem was somewhere deeper down the rabbit hole.
Even software that’s been around for nearly a quarter-century can
harbor some remarkably well-established bugs. Enter
ExtUtils::MakeMaker, which traces its history back to the early days
of Perl’s module-building ecosystem.
Finding this was relatively easy since cpm preserves its build
information, including the generated Makefile. Inspecting the actual
code rather than relying on documentation regarding how .packlist is
generated and taking that as gospel revealed the defect hiding
in plain sight. For the life of me, though, I just don’t understand how
someone with more toolchain chops did not trip over this a long time
ago.
And here it is…the generated Makefile contained the following installation instructions:
read "$(SITEARCHEXP)/auto/$(FULLEXT)/.packlist"
write "$(DESTINSTALLSITEARCH)/auto/$(FULLEXT)/.packlist"
Nothing particularly suspicious until we examined the values of those variables.
The installation had been configured with:
INSTALL_BASE = /home/rlauer
But the relevant directories resolved to:
SITEARCHEXP:
/usr/local/lib/x86_64-linux-gnu/perl/5.40.1
INSTALLSITEARCH:
/home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi
So MakeMaker was attempting to read the previous installation’s packlist from the system Perl directory while writing the updated inventory into my local Perl library.
To be clear, this isn’t merely an unfortunate interaction between installation options. It’s a bug, despite my attempts to convince myself that I’m simply misunderstanding the toolchain.
I suppose one could imagine a design in which Perl maintained a centralized inventory of every distribution installed, regardless of destination. But such a design would require write access to Perl’s system directories, potentially requiring root privileges even for installations into a user’s home directory. That would largely defeat the purpose of local installations. MakeMaker instead writes the packlist locally while attempting to read its history from the system directory. And so I concluded that there is no coherent installation model in which that asymmetry makes any sense whatsoever. And yet, after spending this much time in the Perl toolchain, I’ve learned to question even the conclusions that seem obvious.
Consequently, even though ExtUtils::Install contains code
specifically intended to merge the old inventory with the new one,
that mechanism never received the old inventory.
And that is why files were disappearing from the packlist.
The same path inconsistency also affects MakeMaker’s generated uninstall machinery, which can look in the wrong directory for the inventory it needs to remove.
The correction is relatively straightforward.
When INSTALL_BASE is in effect, MakeMaker should locate the existing
packlist in the effective installation directory, rather than blindly
using Perl’s configured system site directory.
For a site installation, the relevant distinction is:
# Existing behavior
read "$(SITEARCHEXP)/auto/$(FULLEXT)/.packlist"
# Corrected behavior with INSTALL_BASE
read "$(INSTALLSITEARCH)/auto/$(FULLEXT)/.packlist"
The corresponding uninstall lookup needs the same treatment.
A small, focused patch is all that is required.
The PR has been submitted for upstream consideration.
What’s interesting about this adventure is that the old discussion in issue #173 and the bug in #491 are related, but they’re not the same problem.
Issue #173 asks whether an installation inventory should be cumulative in the first place.
Issue #491 demonstrates that MakeMaker’s existing cumulative behavior
wasn’t functioning correctly when installing with INSTALL_BASE.
The former is an architectural debate. The latter is an implementation defect.
And neither changes the fundamental distinction between the contents of a release and the accumulated state of an installation.
Perhaps someday Perl’s installation tooling will support version-specific inventories alongside a cumulative installation history. That would give package builders the clean manifests they want while preserving the information needed to remove obsolete files deliberately.
Until then, MakeMaker’s cumulative packlist behavior serves an important purpose.
Assuming it actually works. Hare’s hoping it will soon.