Part II - filling in the cracks

Part I ended with a fairly simple wish: I want my DarkPAN and public CPAN to participate in dependency resolution together, in an order I control, with access to the full version history of both.

First, order matters. My DarkPAN is not just another place to download a tarball. It may contain private distributions, generated modules, patched releases, or versions that deliberately differ from what is available on public CPAN. If I put that repository first, I am expressing policy: ask it first, and only move on if it cannot satisfy the request.

Second, history matters. If my repository contains versions 2.5.9, 2.6.0, and 2.6.1 of a distribution, I should be able to ask for any of them. Version pinning is not an edge case. It is how we reproduce deployments, roll back regressions, honor compatibility constraints, and keep older applications running against the versions they were built and tested with. A repository that physically contains those releases but can only resolve the newest one is throwing away useful information.

And order matters for performance too. My DarkPAN already contains many of the dependencies my applications use, including distributions pulled in from public CPAN. If the first resolver can answer locally, there is no reason to go out to a public service and ask the same question again. The repository I trust first is often also the fastest place to resolve from.

So the behavior I want is not complicated:

1. Ask my repository first.
2. Let it answer from everything it contains.
3. Honor normal version constraints.
4. If it cannot satisfy the request, move on to the next repository.

In other words, I want a resolver chain made up of independent authorities.

That is where cpm enters the story.

The new kid on the block

cpm is a relatively new installer that improves performance by separating dependency resolution from installation.

Instead of treating resolution as a single built-in service, cpm has the concept of a resolver: something that is given a module name and version requirement and either returns a distribution that satisfies it or declines to answer.

More importantly, resolvers can be composed.

That means the behavior I described above is not something we have to fake with mirror flags. It can be expressed directly:

ask my DarkPAN
    |
    | cannot satisfy the request
    v
ask the next resolver
    |
    v
public CPAN

Each resolver is responsible for answering only for the source it represents. If my DarkPAN knows about Amazon::API, it gets the first opportunity to satisfy the request. If it does not, resolution continues. No resolver has to pretend that every repository contains the same universe of distributions.

But there is still a problem.

A better resolver interface cannot recover information that the repository index never recorded in the first place. If my DarkPAN still describes itself only through 02packages.details.txt.gz, then even the best custom resolver can see only the latest indexed version of each module.

However, cpm gives us two places to plug in the answer!

cpm already points us to the answer

Before going any further, there is an important feature of cpm worth calling out: it already understands that a DarkPAN can have its own resolution service.

The documented DarkPAN support includes the familiar 02packages resolver:

cpm install \
  --resolver 02packages,http://example.com/darkpan \
  Module

but it can also use a MetaDB that you host yourself:

cpm install \
  --resolver metadb,http://example.com/darkmetadb,http://example.com/darkpan \
  Module

That is significant because the model is already much closer to what I described above. The DarkPAN is no longer merely another download location. It can have a resolver that speaks authoritatively about its own contents.

So why didn’t I simply stand up a MetaDB and instead build a custom resolver?

Because doing so would have changed the shape of my repository by requiring the operation and maintenance of a more complicated service whose job is to answer simple resolution requests. My DarkPAN, which had previously been nothing more than static objects served over HTTP, would need to host another moving part: an application endpoint that has to be deployed, monitored, and kept available.

My DarkPAN is a static S3 website with nowhere to run a RESTful API. While I could run a localhost MetaDB service backed by SQLite, it would introduce a daemon simply to implement an HTTP interface that isn’t necessary. Instead, my resolver can just download the index and query it locally.

The interesting thing about cpm’s MetaDB support is that it reinforced the model: a repository should be able to provide its own resolution data and cpm creates the hooks to do so.

The only question left then was what form that implementation should take.

An index of contents, not current state

The missing piece is not another installer and not another mirror flag. It is an index that actually describes what the repository contains.

02packages.details.txt.gz does not do that. It tells us the current winner for each module. That is useful for traditional CPAN mirror semantics, but it throws away exactly the information we now care about: every historical version still present in the repository.

So DarkPAN::Indexer takes a different approach.

It walks the DarkPAN’s authors/id/ hierarchy and examines every distribution tarball it finds. For each distribution, it records every module provided by that distribution and the module version associated with it.

The intermediate representation is deliberately simple:

distribution-tarball<TAB>module-name<TAB>module-version

For example:

Amazon-API-2.5.9.tar.gz    Amazon::API    2.5.9
Amazon-API-2.6.0.tar.gz    Amazon::API    2.6.0
Amazon-API-2.6.1.tar.gz    Amazon::API    2.6.1

If a module does not declare its own version, the distribution version is used instead.

There is no attempt to simply record the newest version. There is no deduplication. There is no PAUSE-style winner calculation. If three releases contain Amazon::API, all three records are preserved.

Why SQLite?

Once DarkPAN::Indexer has preserved the full repository history, the next question is how to make that information cheap to query.

The query itself is simple. I need to ask questions like:

Give me every version of Amazon::API in this repository.

and get back the distribution tarball associated with each one.

That does not require a database server. It does not require an API service. It does not require another daemon to deploy and monitor.

SQLite is enough.

The flat index produced by DarkPAN::Indexer is loaded into a small SQLite database with a table containing:

distribution
module
version

and indexes on the fields used for lookup.

The resulting database is then compressed and published alongside the repository itself:

modules/packages.db.gz

That preserves an important property of the DarkPAN: it remains entirely static. The tarballs and SQLite index are just files served over HTTP.

The distribution tarballs are static objects. The SQLite index is another static object. A client does not need access to S3, a database server, or some private resolution service. It only needs HTTP access to the repository.

SQLite is not the resolver here. It is simply the repository’s memory.

The resolver becomes almost boring

Once the repository can describe its full contents, the resolver itself becomes surprisingly small.

DarkPAN::Resolver::SQLite downloads modules/packages.db.gz, expands it into a temporary SQLite database, and prepares a query for the module being requested:

SELECT version, distribution
  FROM modules
 WHERE module = ?

That returns every version of the module known to this repository.

From there the resolver has three jobs:

1. discard versions that do not satisfy the requested constraint;
2. choose the highest remaining version;
3. return the distribution URL to cpm.

The interesting part is that the resolver does not invent its own version semantics.

Versions are stored in SQLite as text, which means this would be wrong:

1.10.0
1.9.0

if we simply asked SQLite to sort them lexically.

Instead, each candidate version is handed to App::cpm::version. The same machinery cpm uses to understand version requirements is used both to test whether a version satisfies the constraint and to compare candidates.

So a request like:

Amazon::API >= 2.6.0

is resolved using cpm’s own interpretation of that requirement, not a second approximation bolted onto the side.

Now prove it

At this point the architecture is interesting, but architecture is not the goal. The goal is to make the requests that failed in Part 1 work naturally.

Start with the easy case: ask for a module without constraining the version.

cpm install \
  --resolver +DarkPAN::Resolver::SQLite,https://cpan.openbedrock.net/orepan2 \
  Amazon::API

The resolver queries the repository index, finds every version of Amazon::API it knows about, and returns the newest one.

Now ask for an exact historical version:

cpm install \
  --resolver +DarkPAN::Resolver::SQLite,https://cpan.openbedrock.net/orepan2 \
  Amazon::API@2.6.0

There is no switch to a different external resolver service. There is no requirement that 2.6.0 appear in 02packages.details.txt.gz. If Amazon-API-2.6.0.tar.gz exists in this repository and its index says that distribution provides Amazon::API version 2.6.0, the resolver can find it.

Version ranges work for the same reason.

The resolver does not ask, “What is the current version of this module?”

It asks:

What versions of this module does this repository contain,
and which of them satisfy the request?

More importantly, latest, exact-version, and range resolution all follow the same path. There is no hidden change of resolver strategy based on the form of the request.

The repository simply answers from what it contains.

And when it isn’t there

The more interesting case is when my DarkPAN cannot satisfy the request.

That is where resolver composition matters.

Suppose I ask for a module that is not present in my repository at all, or for a version constraint that none of its releases can satisfy. DarkPAN::Resolver::SQLite does not treat that as a fatal error. It simply declines to answer.

Then cpm asks the next resolver in the chain.

Conceptually:

my DarkPAN
    |
    | no satisfying distribution
    v
next resolver
    |
    v
public CPAN

My repository gets the first opportunity to satisfy the request because I have chosen to make it authoritative for the distributions it contains. If it has the dependency, resolution stays there. If it does not, public CPAN remains available as the broader fallback.

This is also where the performance benefit becomes real.

If I have already injected a CPAN dependency into my DarkPAN, the first resolver can satisfy it locally. There is no reason to ask a public resolver about a dependency whose distribution is already sitting in my repository. Over time the DarkPAN becomes not only an authoritative source for my own distributions, but also a fast working set of the dependencies my applications actually use.

And unlike --mirror-only, that preference does not require cutting myself off from public CPAN.

No central authority is proxying version information. Each repository answers for itself.

Bottom line

Part I started with a confusing set of options that failed to express the dependency-resolution model I actually wanted.

I thought I was configuring an ordered set of repositories. What I was really configuring was a fetching list around a resolution model that still assumed one canonical CPAN universe.

The solution was not another flag or combination of flags. The solution was to use an installer that separated the responsibilities cleanly.

Finally, my DarkPAN now supplements CPAN rather than replacing it. Moreover, historical versions can be resolved because the repository index actually remembers them. That’s having our cake and eating it too.

What’s Next

DarkPAN::Indexer was built to solve a specific problem: give a resolver a complete view of what my DarkPAN contains.

But the current resolver is only one possible consumer of that information.

The indexer already separates three concerns that do not have to be coupled:

where distributions come from
how their contents are examined
where the resulting metadata is stored

My current implementation walks an S3-backed DarkPAN and publishes a SQLite database because that fits the architecture I have today: static storage, no server, and a resolver that can query locally.

None of those choices has to be permanent.

A filesystem-backed DarkPAN can be indexed the same way. SQLite could be backed by MySQL, DynamoDB, or some other backend if incremental updates, centralized querying, or concurrent access become more important than publishing a static database file.

More interestingly, module/version resolution is only a small part of what can be learned while ingesting a distribution.

The same pass through the repository could inventory documentation such as POD and README.md, record which distributions are well documented and which are not, and eventually provide enough metadata to build a MetaCPAN-like view of a private repository.

It could also record quality metrics at publication time: test coverage information, dependency counts, documentation coverage, complexity, policy checks, or other measurements produced by the build pipeline.

Those checks probably belong before publication when they are being used to decide whether a release should proceed. But recording the results during ingestion answers a different question:

What did we actually publish?

Over time that becomes a historical record of the repository itself, including improvements and regressions that would otherwise disappear as new releases replace old ones.

So DarkPAN::Indexer may have started as a way to build a better resolver index, but with a little imagination it may become something more: a repository of metadata about your DarkPAN.