Credit: Image generated by Gemini

A couple of months ago, AI and I ventured down a rabbit hole investigating how Perl distributions are located and retrieved from CPAN and private repositories. That adventure resulted in a two-part series, Reflecting on Mirrors. In a somewhat related side tunnel I managed to crack Apache2::Request’s enigmatic build and installation process when using cpm, eventually resulting in patches to both libapreq2 and cpm.

Emerging from the tunnel and basking in the sunlight of discoveries was, apparently, a short-lived reprieve from the mysteries and intricacies of the Perl toolchain.

A 20-year-old question

In my never-ending unintentional discovery of oddities, I bumped into yet another one this past week. The situation revolved around trying to understand why .packlist was losing entries from previous installations.

First, a little history.

Doing a bit of research on the old issues list for ExtUtils::MakeMaker I came across an interesting discussion titled “packlists aren’t pure.” Strap in, turn on your flashlight, we’re going down the tunnel.

To Merge or Not to Merge? That is the question!

The author of the issue questioned MakeMaker’s practice of reading an existing .packlist and merging its contents with the inventory of files being installed in a new version.

He posits, “Why not simply write a new packlist every time, as Module::Build does?”

It’s a reasonable question, and one the designers of ExtUtils::MakeMaker presumably considered. While the implementation doesn’t explain their reasoning, the decision to merge appears deliberate.

Why would you want to preserve the list of files in every distribution? The immediate answer is revealed when you ask: How can I remove all of the files installed by a distribution?

Consider a distribution that evolves through three releases:

Release Files included
1.0 foo.xml, bar.xml
1.1 bar.xml
1.2 baz.xml

If we merge successive installation inventories, the resulting packlist contains all three files.

If we replace the inventory on each installation, the packlist contains only baz.xml.

Which behavior is correct?

Well, that depends on what you believe a packlist is supposed to represent.

Installation Is Easy. Uninstallation? Not So Much.

Installing a Perl distribution is a relatively straightforward affair. The distribution declares its dependencies, an installer resolves them, and the files end up somewhere you specified-hopefully somewhere Perl can find them.

Uninstalling that distribution is another matter entirely.

For starters, Perl doesn’t maintain the kind of centralized package database found in RPM or Debian-based systems. A typical Perl application may depend on modules supplied by the operating system, modules installed from CPAN, and modules vendored locally with the application.

Perl’s @INC determines where modules are found, but it tells us nothing about who installed them or who might still need them.

Then there’s the question of file ownership. Suppose distribution A installs my favorite script, foo.pl, and distribution B subsequently installs a script with the same name into the same directory. If we uninstall A, should foo.pl disappear?

And what happens if B depends on a module supplied by A?

These are legitimate concerns that have led to a rather curious solution. Using ExtUtils::MakeMaker:

…you can generate a Makefile and actually type:

make uninstall

But you may be surprised to discover that nothing actually gets uninstalled.

MakeMaker deliberately invokes ExtUtils::Install::uninstall() in dry-run mode, warning that uninstallation is unsafe and deprecated. Huh? And cpanm lists the --uninstall option as experimental.

The underlying ExtUtils::Install function can in fact remove the files. MakeMaker simply chooses not to let its generated target (uninstall) do so. I suppose that is a reasonable approach if your goal is to make sure your footgun doesn’t fire.

And there’s another oddity: to invoke make uninstall, you need a generated Makefile. That means retaining the build directory or obtaining the distribution and generating its Makefile again, just to attempt to uninstall something that’s already installed!

And as I said, cpanm --uninstall provides an alternative that works directly from the installed distribution’s packlist, although the command is mysteriously labeled experimental as if we’re waiting on the results for something(?) to decide if it stays or if it goes.

I find both of these approaches a bit paternalistic. Dependency and ownership risks are reasons to warn users about uninstallation, not necessarily reasons to call it experimental or prevent it from actually working.

But perhaps there’s another reason cpanm labeled the feature experimental. What if the inventory used to uninstall a distribution isn’t reliable in the first place? Spoiler alert…it isn’t.

That brings us back to the humble .packlist.

If we’re going to remove everything a distribution installed, we first need to know what it installed. Not just today, but throughout its installation history.

And that is why the question of merging versus replacing packlists matters.

A release manifest is not an installation history

The argument in #173 was largely motivated by the needs of package maintainers. When constructing an RPM or Debian package, the packager wants an accurate inventory of the files belonging to that particular release.

Files from earlier releases don’t belong in that inventory.

Fair enough.

But installing a Perl distribution isn’t quite the same thing as installing an RPM or Debian package.

Operating-system package managers maintain centralized databases of installed packages, file ownership, versions, dependencies, and removal operations. They have considerable machinery dedicated to managing a package’s lifecycle.

Why is Perl different?

Perl distributions don’t have an equivalent centralized ownership database.

There’s a fundamental difference between operating-system package managers like RPM and dpkg and language-specific installers like cpanm, cpm, or Python’s pip.

RPM and Debian packages are primarily intended to install and manage software within an operating system. Yes, alternative installation roots and other arrangements are possible, but that’s not how most application developers use them, assuming they use them at all. These tools generally operate within a centrally managed environment, frequently requiring root privileges to install or remove system packages.

Perl and Python inhabit something closer to the Wild West. Developers routinely install modules into their home directories, application-specific directories, virtual environments, or vendored dependency trees. New little towns spring up everywhere, each with its own collection of packages, and none necessarily aware of the others.

Perl compounds the situation by maintaining separate directories for core, vendor, and site modules. Add local::lib and application-vendored modules, and it’s not unusual for a single application to resolve dependencies from three or four different installation trees.

Perl knows where to look through @INC, but it doesn’t maintain a centralized registry of who installed what, where, or who still depends on it.

That makes the humble .packlist considerably more important than it might first appear. For an independently managed Perl installation, it may be the only enduring record of the files a distribution has installed.

Unlike an RPM or Debian package, a Perl distribution cannot assume that some other package manager is keeping track of the files it leaves behind.

And that’s where replacing a cumulative packlist becomes problematic.

Suppose version 1.0 installed foo.xml, but version 1.1 dropped it. If installation does not remove obsolete files, foo.xml is still sitting on disk.

If the new installation also removes foo.xml from the packlist, we’ve now lost track of a file we installed.

A subsequent uninstall cannot remove a file it no longer knows exists.

The two requirements are different:

  • A release inventory describes what a particular version contains.
  • An installation inventory describes what has been installed over the lifetime of a distribution.

Trying to make one inventory satisfy both requirements is where the trouble begins.

A Better Mousetrap?

One could imagine versioned packlists:

.packlist.3.9.2
.packlist.3.9.3
.packlist.3.9.4

Each would provide an accurate inventory of a particular release. A complete uninstall could examine the accumulated inventories and remove the union of files installed by the distribution.

That would provide both a clean release inventory and a historical record.

Of course, this is an architectural change. Existing Perl tooling expects a conventional .packlist, and introducing versioned inventories would raise questions about backward compatibility, file ownership, and uninstallation semantics.

Before anyone thinks I’m proposing a redesign of MakeMaker’s installation machinery to solve a potential problem, let me assure you I’m not!

For now, its existing cumulative packlist behavior seems a reasonable compromise.

Assuming it actually worked.

Well does it? Let’s go down the rabbit hole…

Let’s look at a real-life case where we can see if ExtUtils::MakeMaker actually does what it thinks it does.

Consider what happend when I restructured a CPAN distribution, moving functionality into more narrowly focused distributions. As part of that work, files were relocated or removed from the new release.

That raised an obvious question: what happens to those files in the older releases when we install the new releases of the now split package?

We don’t particularly want installations silently deleting files from earlier releases during installation. Installation and uninstallation should be distinct operations, with the latter deliberately initiated by someone who knows what they’re removing.

And…ExtUtils::MakeMaker’s implementation of make uninstall which uses .packlist looks like it might help to uninstall files…but runs in dry mode!

As we discovered earlier, MakeMaker’s make uninstall doesn’t actually uninstall anything. Fortunately, ExtUtils::Install provides the underlying operation directly:

use ExtUtils::Install;

ExtUtils::Install::uninstall(
    '/path/to/auto/Bedrock/Web/.packlist',
    1,  # verbose
    1,  # dry run - change me to 0 to do the deed
);

So does it actually work? Well, yes and no. It works if you are installing things into the system directory, but not when you use INSTALL_BASE. Let me explain.

While examining the packlist for my newly split distribution I noticed that files removed from this new release were disappearing from the inventory after reinstalling the distribution into my local environment.

The files were no longer recorded in .packlist as they should have been, even though they hadn’t been uninstalled.

The experiment

Install the original version of a distribution containing some file and inspect its packlist:

grep bedrock\.pl /home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Bedrock/Web/.packlist
/home/rlauer/bin/bedrock.pl

Next, remove the file from the distribution, rebuild it, and reinstall it.

grep bedrock\.pl /home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Bedrock/Web/.packlist

Gone. Vanished. Poof! Hey! Where did it go?

Not necessarily from the filesystem, only from .packlist. And bedrock.pl is now forever orphaned like a lost rabbit.

command -v bedrock.pl
/home/rlauer/bin/bedrock.pl

And that was precisely the behavior MakeMaker’s merging mechanism was supposed to prevent.

Round up the usual suspects

Could it be the installer itself? My first suspicion fell on cpm since I use that as my installer these days. I knew that it used its own installation machinery by default.

Perhaps it was simply replacing the packlist?

So I tried again with --use-install-command, directing cpm to execute the distribution’s installation command.

Same result.

So I repeated the experiment using cpanm.

Same result.

At that point, either both installers were making the same mistake, or the problem was somewhere deeper down the rabbit hole.

The smoking gun

Even software that’s been around for nearly a quarter-century can harbor some remarkably well-established bugs. Enter ExtUtils::MakeMaker, which traces its history back to the early days of Perl’s module-building ecosystem.

Finding this was relatively easy since cpm preserves its build information, including the generated Makefile. Inspecting the actual code rather than relying on documentation regarding how .packlist is generated and taking that as gospel revealed the defect hiding in plain sight. For the life of me, though, I just don’t understand how someone with more toolchain chops did not trip over this a long time ago.

And here it is…the generated Makefile contained the following installation instructions:

read "$(SITEARCHEXP)/auto/$(FULLEXT)/.packlist"
write "$(DESTINSTALLSITEARCH)/auto/$(FULLEXT)/.packlist"

Nothing particularly suspicious until we examined the values of those variables.

The installation had been configured with:

INSTALL_BASE = /home/rlauer

But the relevant directories resolved to:

SITEARCHEXP:
/usr/local/lib/x86_64-linux-gnu/perl/5.40.1

INSTALLSITEARCH:
/home/rlauer/lib/perl5/x86_64-linux-gnu-thread-multi

So MakeMaker was attempting to read the previous installation’s packlist from the system Perl directory while writing the updated inventory into my local Perl library.

To be clear, this isn’t merely an unfortunate interaction between installation options. It’s a bug, despite my attempts to convince myself that I’m simply misunderstanding the toolchain.

I suppose one could imagine a design in which Perl maintained a centralized inventory of every distribution installed, regardless of destination. But such a design would require write access to Perl’s system directories, potentially requiring root privileges even for installations into a user’s home directory. That would largely defeat the purpose of local installations. MakeMaker instead writes the packlist locally while attempting to read its history from the system directory. And so I concluded that there is no coherent installation model in which that asymmetry makes any sense whatsoever. And yet, after spending this much time in the Perl toolchain, I’ve learned to question even the conclusions that seem obvious.

Consequently, even though ExtUtils::Install contains code specifically intended to merge the old inventory with the new one, that mechanism never received the old inventory.

And that is why files were disappearing from the packlist.

The same path inconsistency also affects MakeMaker’s generated uninstall machinery, which can look in the wrong directory for the inventory it needs to remove.

A small fix for a longstanding problem

The correction is relatively straightforward.

When INSTALL_BASE is in effect, MakeMaker should locate the existing packlist in the effective installation directory, rather than blindly using Perl’s configured system site directory.

For a site installation, the relevant distinction is:

# Existing behavior
read "$(SITEARCHEXP)/auto/$(FULLEXT)/.packlist"

# Corrected behavior with INSTALL_BASE
read "$(INSTALLSITEARCH)/auto/$(FULLEXT)/.packlist"

The corresponding uninstall lookup needs the same treatment.

A small, focused patch is all that is required.

The PR has been submitted for upstream consideration.

Emerging into the sunlight once more

What’s interesting about this adventure is that the old discussion in issue #173 and the bug in #491 are related, but they’re not the same problem.

Issue #173 asks whether an installation inventory should be cumulative in the first place.

Issue #491 demonstrates that MakeMaker’s existing cumulative behavior wasn’t functioning correctly when installing with INSTALL_BASE.

The former is an architectural debate. The latter is an implementation defect.

And neither changes the fundamental distinction between the contents of a release and the accumulated state of an installation.

Perhaps someday Perl’s installation tooling will support version-specific inventories alongside a cumulative installation history. That would give package builders the clean manifests they want while preserving the information needed to remove obsolete files deliberately.

Until then, MakeMaker’s cumulative packlist behavior serves an important purpose.

Assuming it actually works. Hare’s hoping it will soon.

Links


Previous post: Simple is Not Lite - CLI::Simple: A Robust Scripting Framework