nginx is dead. Not metaphorically dead. Not “falling out of favor” dead. Actually, officially, put-a-date-on-it dead.

In November 2025 the Kubernetes project announced the retirement of Ingress NGINX — the controller running ingress for a significant fraction of the world’s Kubernetes clusters. Best-effort maintenance until March 2026. After that: no releases, no bugfixes, no security patches. GitHub repositories go read-only. Tombstone in place.

And before the body was even cold, we learned why. IngressNightmare — five CVEs disclosed in March 2025, headlined by CVE-2025-1974, rated 9.8 critical. Unauthenticated remote code execution. Complete cluster takeover. No credentials required. Wiz Research found over 6,500 clusters with the vulnerable admission controller publicly exposed to the internet, including Fortune 500 companies. 43% of cloud environments vulnerable. The root cause wasn’t a bug that could be patched cleanly - it was an architectural flaw baked into the design from the beginning. And the project that ran ingress for millions of production clusters was, in the end, sustained by one or two people working in their spare time.

Meanwhile Apache has been quietly running the internet for 30 years, governed by a foundation, maintained by a community, and looking increasingly like the adult in the room.

Let’s talk about how we got here.

Apache Was THE Web Server

Before we talk about what went wrong, let’s remember what Apache actually was. Not a web server. THE web server. At its peak Apache served over 70% of all websites on the internet. It didn’t win that position by accident - it won it by solving every problem the early web threw at it. Virtual hosting. SSL. Authentication. Dynamic content via CGI and then mod_perl. Rewrite rules. Per-directory configuration. Access control. Compression. Caching. Proxying. One by one, as the web evolved, Apache evolved with it, and the industry built on top of it.

Apache wasn’t just infrastructure. It was the platform on which the commercial internet was built. Every hosting provider ran it. Every enterprise deployed it. Every web developer learned it. It was as foundational as TCP/IP - so foundational that most people stopped thinking about it, the way you stop thinking about running water.

Then nginx showed up with a compelling story at exactly the right moment.

The Narrative That Stuck

The early 2000s brought a new class of problem - massively concurrent web applications, long-polling, tens of thousands of simultaneous connections. The C10K problem was real and Apache’s prefork MPM - one process per connection - genuinely struggled under that specific load profile. nginx’s event-driven architecture handled it elegantly. The benchmarks were dramatic. The config was clean and minimal, a breath of fresh air compared to Apache’s accumulated complexity. nginx felt modern. Apache felt like your dad’s car.

The “Apache is legacy” narrative took hold and never let go - even after the evidence for it evaporated.

Apache gained mpm_event, bringing the same non-blocking I/O and async connection handling that nginx was celebrated for. The performance gap on concurrent connections essentially closed. Then CDNs solved the static file problem at the architectural level - your static files live in S3 now, served from a Cloudflare edge node milliseconds from your user, and your web server never sees them. The two pillars of the nginx argument - concurrency and static file performance - were addressed, one by Apache’s own evolution and one by infrastructure that any serious deployment should be using regardless of web server choice.

But nobody reruns the benchmarks. The “legacy” label outlived the evidence by a decade. A generation of engineers learned nginx first, taught it to the next generation, and the assumption calcified into received wisdom. Blog posts from 2012 are still being cited as architectural guidance in 2025.

What Apache Does That nginx Can’t

Strip away the benchmark mythology and look at what these servers actually do when you need them to do something hard.

Apache’s input filter chain lets you intercept the raw request byte stream mid-flight - before the body is fully received - and do something meaningful with it. I’m currently building a multi-server file upload handler with real-time Redis progress tracking, proper session authentication, and CSRF protection implemented directly in the filter chain. Zero JavaScript upload libraries. Zero npm dependencies. Zero supply chain attack surface. The client sends bytes. Apache intercepts them. Redis tracks them. Done. nginx needs a paid commercial module to get close. Or you write C. Or you route around it to application code and wonder why you needed nginx in the first place.

Apache’s phase handlers let you hook into the exact right moment of the request lifecycle - post-read, header parsing, access control, authentication, response - each phase a precise intervention point. mod_perl embeds a full Perl runtime in the server with persistent state, shared memory, and pre-forked workers inheriting connection pools and compiled code across requests. mod_security gives you WAF capabilities your “modern” stack is paying a vendor for. mod_cache is a complete RFC-compliant caching layer that nginx reserves for paying customers.

And LDAP - one of the oldest enterprise authentication requirements there is. With mod_authnz_ldap it’s a few lines of config:

AuthType Basic
AuthName "Corporate Login"
AuthBasicProvider ldap
AuthLDAPURL ldap://ldap.company.com/dc=company,dc=com
AuthLDAPBindDN "cn=apache,dc=company,dc=com"
AuthLDAPBindPassword secret
Require ldap-group cn=developers,ou=groups,dc=company,dc=com

Connection pooling, SSL/TLS to the directory, group membership checks, credential caching - all native, all in config, no code required. With nginx you’re reaching for a community module with an inconsistent maintenance history, writing Lua, or standing up a separate auth service and proxying to it with auth_request - which is just mod_authnz_ldap reimplemented badly across two processes with an HTTP round trip in the middle.

Apache Includes Everything You’re Now Paying For

Look at Apache’s feature set and you’re reading the history of web infrastructure, one solved problem at a time. SSL termination? Apache had it before cloud load balancers existed to take it off your plate. Caching? mod_cache predates Redis by years. Load balancing? mod_proxy_balancer was doing weighted round-robin and health checks before ELB was a product. Compression, rate limiting, IP-based access control, bot detection via mod_security - Apache had answers to all of it before the industry decided each problem deserved its own dedicated service, its own operations overhead, and its own vendor relationship.

Apache didn’t accumulate features because it was undisciplined. It accumulated features because the web kept throwing problems at it and it kept solving them. The fact that your load balancer now handles SSL termination doesn’t mean Apache was wrong to support it - it means Apache was right early enough that the rest of the industry eventually built dedicated infrastructure around the same idea.

Now look at your AWS bill. CloudFront for CDN. ALB for load balancing and SSL termination. WAF for request filtering. ElastiCache for caching. Cognito for authentication. API Gateway for routing. Each one a line item. Each one a managed service wrapping functionality that Apache has shipped for free since before most of your team was writing code.

Amazon Web Services is, in a very real sense, Apache’s feature set repackaged as paid managed infrastructure. They looked at what the web needed, looked at what Apache had already solved, and built a business around operating those solutions at scale so you didn’t have to. That’s a legitimate value proposition - operations is hard and sometimes paying AWS is absolutely the right answer. But if you’re running a handful of servers and paying for half a dozen AWS services to handle concerns that Apache handles natively, maybe set the Wayback Machine to 2005, spin up Apache, and keep the credit card in your pocket.

Grandpa wasn’t just ahead of his time. Grandpa was so far ahead that Amazon built a cloud business catching up to him.

So Why Did You Choose nginx?

Be honest. The real reason is that you learned it first, or your last job used it, or a blog post from 2012 told you it was the modern choice. Maybe someone at a conference said Apache was legacy and you nodded along because everyone else was nodding. That’s how technology adoption works - narrative momentum, not engineering analysis.

But those nginx blinders have a cost. And the Kubernetes ecosystem just paid it in full.

The Cost of the nginx Blinders

The nginx Ingress Controller became the Kubernetes default early in the ecosystem’s adoption curve and the pattern stuck. Millions of production clusters. The de-facto standard. Fortune 500 companies. The Swiss Army knife of Kubernetes networking - and that flexibility was precisely its undoing.

The “snippets” feature that made it popular - letting users inject raw nginx config via annotations - turned out to be an unsanitizable attack surface baked into the design. CVE-2025-1974 exploited this to achieve unauthenticated RCE via the admission controller, giving attackers access to all secrets across all namespaces. Complete cluster takeover from anything on the pod network. In many common configurations the pod network is accessible to every workload in your cloud VPC. The blast radius was the entire cluster.

The architectural flaw couldn’t be fixed without gutting the feature that made the project worth using. So it was retired instead.

Here is the part nobody is saying out loud: Apache could have been your Kubernetes ingress controller all along.

The Apache Ingress Controller exists. It supports path and host-based routing, TLS termination, WebSocket proxying, header manipulation, rate limiting, mTLS - everything Ingress NGINX offered, built on a foundation with 30 years of security hardening and a governance model that doesn’t depend on one person’s spare time. It doesn’t have an unsanitizable annotation system because Apache’s configuration model was designed with proper boundaries from the beginning. The full Apache module ecosystem - mod_security, mod_authnz_ldap, the filter chain, all of it - available to every ingress request.

The Kubernetes community never seriously considered it. nginx had the mindshare, nginx got the default recommendation, nginx became the assumed answer before the question was even finished. Apache was dismissed as grandpa’s web server by engineers who had never actually used it for anything hard - and so the ecosystem bet its ingress layer on a project sustained by volunteers and crossed its fingers.

The nginx blinders cost the industry IngressNightmare, 6,500 exposed clusters, and a forced migration that will consume engineering hours across thousands of organizations in 2026. Not because Apache wasn’t available. Because nobody looked.

nginx is survived by its commercial fork nginx Plus, approximately 6,500 vulnerable Kubernetes clusters, and a generation of engineers who will spend Q1 2026 migrating to Gateway API - a migration they could have avoided entirely.

Who’s Keeping The Lights On

Here’s the conversation that should happen in every architecture review but almost never does: who maintains this and what happens when something goes wrong?

For Apache the answer has been the same for over 30 years. The Apache Software Foundation - vendor-neutral, foundation-governed, genuinely open source. Security vulnerabilities found, disclosed responsibly, patched. A stable API that doesn’t break your modules between versions. Predictable release cycles. Institutional stability that has outlasted every company that ever tried to compete with it.

nginx’s history is considerably more complicated. Written by Igor Sysoev while employed at Rambler, ownership murky for years, acquired by F5 in 2019. Now a critical piece of infrastructure owned by a networking hardware vendor whose primary business interests may or may not align with the open source project. nginx Plus - the version with the features that actually compete with Apache on a level playing field - is commercial. OpenResty, the variant most people reach for when they need real programmability, is a separate project with its own maintenance trajectory.

The Ingress NGINX project had millions of users and a maintainership you could count on one hand. That’s not a criticism of the maintainers - it’s an indictment of an ecosystem that adopted a critical infrastructure component without asking who was keeping the lights on.

Three decades of adversarial testing by the entire internet is a security posture no startup’s stack can match. The Apache Software Foundation will still be maintaining Apache httpd when the company that owns your current stack has pivoted twice and been acqui-hired into oblivion.

Long Live Apache

The engineers who dismissed Apache as legacy were looking at a 2003 benchmark and calling it a verdict. They missed the server that anticipated every problem modern infrastructure is still solving, that powered the internet before AWS existed to charge you for the privilege, and that was sitting right there in the Kubernetes ecosystem waiting to be evaluated while the community was busy betting critical infrastructure on a volunteer project with an architectural time bomb in its most popular feature.

Grandpa didn’t just know what he was doing. Grandpa was building the platform you’re still trying to reinvent - badly, in JavaScript, with a vulnerability disclosure coming next Tuesday and a maintainer burnout announcement the Tuesday after that.

The server is fine. It was always fine. Touch grass, update your mental model, and maybe read the Apache docs before your next architecture meeting.

RIP nginx Ingress Controller. 2015-2026. Maintained by one guy in his spare time. Missed by the 43% of cloud environments that probably should have asked more questions.

Sources

• IngressNightmare - CVE details and exposure statistics Wiz Research, March 24, 2025 https://www.wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities
• Ingress NGINX Retirement Announcement Kubernetes SIG Network and Security Response Committee, November 11, 2025 https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/
• Ingress NGINX CVE-2025-1974 - Official Kubernetes Advisory Kubernetes, March 24, 2025 https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/
• Transitioning Away from Ingress NGINX - Maintainership and architectural analysis Google Open Source Blog, February 2026 https://opensource.googleblog.com/2026/02/the-end-of-an-era-transitioning-away-from-ingress-nginx.html
• F5 Acquisition of nginx F5 Press Release, March 2019 https://www.f5.com/company/news/press-releases/f5-acquires-nginx-to-bridge-netops-and-devops

Disclaimer: This article was written with AI assistance during a long discussion on the features and history of Apache and nginx, drawing on my experience maintaining and using Apache over the last 20+ years. The opinions, technical observations, and arguments are entirely my own. I am in no way affiliated with the ASF, nor do I have any financial interest in promoting Apache. I have been using and benefiting from Apache since 1998 and continue to discover features and capabilities that surprise me even to this day.

This is the last in a 3 part series on Scriptlets. You can catch up by reading our introduction and dissection of Scriptlets.

• Part I - An introduction to scriptlets
• Part II - The anatomy of a scriptlet

In this final part, we talk about restraint - the discipline that keeps a clever trick from turning into a maintenance hazard.

That uneasy feeling…

So you are starting to write a few scriptlets and it seems pretty cool. But something doesn’t feel quite right…

You’re editing a Makefile and suddenly you feel anxious. Ah, you expected syntax highlighting, linting, proper indentation, and maybe that warm blanket of static analysis. So when we drop a 20 - line chunk of Perl or Python into our Makefile, our inner OCD alarms go off. No highlighting. No linting. Just raw text.

The discomfort isn’t a flaw - it’s feedback. It tells you when you’ve added too much salt to the soup.

A scriptlet is not a script!

A scriptlet is a small, focused snippet of code embedded inside a Makefile that performs one job quickly and deterministically. The “-let” suffix matters. It’s not a standalone program. It’s a helper function, a convenience, a single brushstroke that belongs in the same canvas as the build logic it supports.

If you ever feel the urge to bite your nails, pick at your skin, or start counting the spaces in your indentation - stop. You’ve crossed the line. What you’ve written is no longer a scriptlet; it’s a script. Give it a real file, a shebang, and a test harness. Keep the build clean.

Why we use them

Scriptlets shine where proximity and simplicity matter more than reuse (not that we can’t throw it in a separate file and include it in our Makefile).

• Cleanliness: prevents a recipe from looking like a shell script.
• Locality: live where they’re used. No path lookups, no installs.
• Determinism: transform well-defined input into output. Nothing more.
• Portability (of the idea): every CI/CD system that can run make can run a one-liner.

A Makefile that can generate its own dependency file, extract version numbers, or rewrite a cpanfile doesn’t need a constellation of helper scripts. It just needs a few lines of inline glue.

Why they’re sometimes painful

We lose the comforts that make us feel like professional developers:

• No syntax highlighting.
• No linting or type hints.
• No indentation guides.
• No “Format on Save.”

The trick is to accept that pain as a necessary check on the limits of the scriptlet. If you’re constantly wishing for linting and editor help, it’s your subconscious telling you: this doesn’t belong inline anymore. You’ve outgrown the -let.

When to promote your scriplet to a script…

Promote a scriptlet to a full-blown script when:

• It exceeds 30-50 lines.
• It gains conditionals or error handling.
• You need to test it independently.
• It uses more than 1 or 2 non-core features.
• It’s used by more than one target or project.
• You’re debugging quoting more than logic.
• You’re spending more time fixing indentation, than working on the build

At that point, you’re writing software, not glue. Give it a name, a shebang, and a home in your tools/ directory.

When to keep it inside your Makefile…

Keep it inline when:

• It’s short, pure, and single-use.
• It depends primarily on the environment already assumed by your build (Perl, Python, awk, etc.).
• It’s faster to read than to reference.

A good scriptlet reads like a make recipe: do this transformation right here, right now.

define create_cpanfile =
    while (<STDIN>) {
        s/[#].*//; s/^\s+|\s+$//g; next if $_ eq q{};
        my ($mod,$v) = split /\s+/, $_, 2;
        print qq{requires "$mod", "$v";\n};
    }
endef

export s_create_cpanfile = $(value create_cpanfile)

That’s a perfect scriptlet: small, readable, deterministic, and local.

Rule of Thumb: If it fits on one screen, keep it inline. If it scrolls, promote it.

Tools for the OCD developer

If you must relieve the OCD symptoms without promotion of your scriptlet to a script…

• Add a lint-scriptlets target: perl -c -e '$(s_create_requires)' checks syntax without running it.
• Some editors (Emacs mmm-mode, Vim polyglot) can treat marked sections as sub-languages to enable localized language specific editing features.
• Use include to include a scriptlet into your Makefile

…however try to resist the urge to over-optimize the tooling. Feeling the uneasiness grow helps identify the boundary between scriptlets and scripts.

You’ve been warned!

Because scriptlets are powerful, flexible, and fast, it’s easy to reach for them too often or make them the focus of your project. They start as a cure for friction - a way to express a small transformation inline - but left unchecked, they can sometimes grow arms and legs. Before long, your Makefile turns into a Frankenstein monster.

The great philosopher Basho (or at least I think it was him) once said:

A single aspirin tablet eases pain. A whole bottle sends you to the hospital.

Thanks for reading.

Learn More

• GNU make
• value
• define
• export

In our previous blog post “Go Ahead ‘make’ My Day” we presented the scriptlet, an advanced make technique for spicing up your Makefile recipes. In this follow-up, we’ll deconstruct the scriptlet and detail the ingredients that make up the secret sauce.

Introducing the Scriptlet

Makefile scriptlets are an advanced technique that uses GNU make’s powerful functions to safely embed a multi-line script (Perl, in our example) into a single, clean shell command. It turns a complex block of logic into an easily executable template.

An Example Scriptlet

#-*- mode: makefile; -*-

DARKPAN_TEMPLATE="https://cpan.openbedrock.net/orepan2/authors/D/DU/DUMMY/%s-%s.tar.gz"

define create_requires =
 # scriptlet to create cpanfile from an list of required Perl modules
 # skip comments
 my $DARKPAN_TEMPLATE=$ENV{DARKPAN_TEMPLATE};

 while (s/^#[^\n]+\n//g){};

 # skip blank lines
 while (s/\n\n/\n/) {};

 for (split/\n/) { 
  my ($mod, $v) = split /\s+/;
  next if !$mod;

  my $dist = $mod;
  $dist =~s/::/\-/g;

  my $url = sprintf $DARKPAN_TEMPLATE, $dist, $v;

  print <<"EOF";
requires \"$mod\", \"$v\",
  url => \"$url\";
EOF
 }

endef

export s_create_requires = $(value create_requires)

cpanfile.darkpan: requires.darkpan
    DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE); \
    DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl -0ne "$$s_create_requires" $< > $@ || rm $@

Dissecting the Scriptlet

1. The Container: Defining the Script (define / endef)

This section creates the multi-line variable that holds your entire Perl program.

define create_requires =
# Perl code here...
endef

• define ... endef: This is GNU Make’s mechanism for defining a recursively expanded variable that spans multiple lines. The content is not processed by the shell yet; it’s simply stored by make.
• The Advantage: This is the only clean way to write readable, indented code (like your while loop and if statements) directly inside a Makefile.

2. The Bridge: Passing Environment Data (my $ENV{...})

This is a critical step for making your script template portable and configurable.

my $DARKPAN_TEMPLATE=$ENV{DARKPAN_TEMPLATE};

• The Problem: Your Perl script needs dynamic values (like the template URL) that are set by make.
• The Solution: Instead of hardcoding the URL, the Perl code is designed to read from the shell environment variable $ENV{DARKPAN_TEMPLATE}. This makes the script agnostic to its calling environment, delegating the data management back to the Makefile.

3. The Transformer: Shell Preparation (export and $(value))

This is the “magic” that turns the multi-line Make variable into a single, clean shell command.

export s_create_requires = $(value create_requires)

• $(value create_requires): This is a specific Make function that performs a direct, single-pass expansion of the variable’s raw content. Crucially, it converts the entire multi-line block into a single-line string suitable for export, preserving special characters and line breaks that the shell will execute.
• export s_create_requires = ...: This exports the multi-line Perl script content as an environment variable (s_create_requires) that will be accessible to any shell process running in the recipe’s environment.

4. The Execution: Atomic Execution ($$ and perl -0ne)

The final recipe executes the entire, complex process as a single, atomic operation, which is the goal of robust Makefiles.

cpanfile.darkpan: requires.darkpan
    DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE); \
    DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl -0ne "$$s_create_requires" $< > $@ || rm $@

• DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE): This creates the local shell variable.
• DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl...: This is the clean execution. The first DARKPAN_TEMPLATE= passes the newly created shell variable’s value as an environment variable to the perl process. The $$ ensures the shell variable is properly expanded before the Perl interpreter runs it.
• perl -0ne "...": Runs the Perl script:

  * `-n` and `-e` (Execute script on input)
  * `-0`: Tells Perl to read the input as one single block
    (slurping the file), which is necessary for your multi-line
    regex and `split/\n/` logic.
  

• || rm $@: This is the final mark of quality. It makes the entire command transactional—if the Perl script fails, the half-written target file ($@) is deleted, forcing make to try again later.

Hey Now! You’re a Rockstar!

(..get your game on!)

Mastering build automation using make will transform you from being an average DevOps engineer into a rockstar. GNU make is a Swiss Army knife with more tools than you might think! The knives are sharp and the tools are highly targeted to handle all the real-world issues build automation has encountered over the decades. Learning to use make effectively will put you head and shoulders above the herd (see what I did there? 😉).

Calling All Pythonistas!

The scriptlet technique creates a powerful, universal pattern for clean, atomic builds:

• It’s Language Agnostic: Pythonistas! Join the fun! The same define/export technique works perfectly with python -c.
• The Win: This ensures that every developer - regardless of their preferred language - can achieve the same clean, atomic build and avoid external script chaos.

Learn more about GNU make and move your Makefiles from simple shell commands to precision instruments of automation.

Thanks for reading.

Learn More

• GNU make
• value
• define
• export

While looking at some old bash script that bumps my semantic versions I almost puked looking at my old ham handed way of bumping the version. That led me to see how I could do it “better”. Why? I dunno know…bored on a Saturday morning and not motivated enough to do the NY Times crossword…

So you want to bump a semantic version string like 1.2.3 - major, minor, or patch - and you don’t want ceremony. You want one line, no dependencies, and enough arcane flair to scare off coworkers.

Here’s a single-line Bash–Perl spell that does exactly that:

v=$(cat VERSION | p=$1 perl -a -F[.] -pe \
'$i=$ENV{p};$F[$i]++;$j=$i+1;$F[$_]=0 for $j..2;$"=".";$_="@F"')

What It Does

• Reads the current version from a VERSION file (1.2.3)
• Increments the part you pass (0 for major, 1 for minor, 2 for patch)
• Resets all lower parts to zero
• Writes the result to v

Scriptlet Form

Wrap it like this in a shell function:

bump() {
  v=$(cat VERSION | p=$1 perl -a -F[.] -pe \
  '$i=$ENV{p};$F[$i]++;$j=$i+1;$F[$_]=0 for $j..2;$"=".";$_="@F"')
  echo "$v" > VERSION
}

Then run:

bump 2   # bump patch (1.2.3 => 1.2.4)
bump 1   # bump minor (1.2.3 => 1.3.0)
bump 0   # bump major (1.2.3 => 2.0.0)

Makefile Integration

Want to bump right from make?

bump-major:
    @v=$$(cat VERSION | p=0 perl -a -F[.] -pe '$$i=$$ENV{p};$$F[$$i]++;$$j=$$i+1;$$F[$$_]=0 for $$j..2;$$"=".";$_="$$F"') && \
    echo $$v > VERSION && echo "New version: $$v"

bump-minor:
    @$(MAKE) bump-major p=1

bump-patch:
    @$(MAKE) bump-major p=2

Or break it out into a .bump-version script and source it from your build tooling.

How It Works (or …Why I Love Perl)

-a            # autosplit into @F
-F[.]         # split on literal dot
$i=$ENV{p}    # get part index from environment (e.g., 1 for minor)
$F[$i]++      # bump it
$j=$i+1       # start index for resetting
$F[$_]=0 ...  # zero the rest
$"=".";       # join array with dots
$_="@F"       # set output

If you have to explain this to some junior dev, just say RTFM skippy perldoc perlrun. Use the force Luke.

And if the senior dev wags his finger and say UUOC, tell him Ego malum edo.

An "END" code block is executed as late as possible, that is, after perl
has finished running the program and just before the interpreter is
being exited, even if it is exiting as a result of a die() function.
(But not if it's morphing into another program via "exec", or being
blown out of the water by a signal--you have to trap that yourself (if
you can).) You may have multiple "END" blocks within a file--they will
execute in reverse order of definition; that is: last in, first out
(LIFO). "END" blocks are not executed when you run perl with the "-c"
switch, or if compilation fails....

Perl’s END blocks are useful inside your script for doing things like cleaning up after itself, closing files or disconnecting from databases. In many cases you use an END block to guarantee certain behaviors like a commit or rollback of a transaction. You’ll typically see END blocks in scripts, but occassionally you might find one in a Perl module.

Over the last four years I’ve done a lot of maintenance work on legacy Perl applications. I’ve learned more about Perl in these four years than I learned in the previous 20. Digging into bad code is the best way to learn how to write good code. It’s sometimes hard to decide if code is good or bad but to paraphrase a supreme court justice, I can’t always define bad code, but I know it when I see it.

One of the gems I’ve stumbled upon was a module that provided needed functionality for many scripts that included its own END block.

Putting an END block in a Perl module is an anti-pattern and just bad mojo. A module should never contain an END block. Here are some alternatives:

• If cleanup is necessary, provide (and document) a cleanup() method
• Use a destructor (DESTROY) in an object-oriented module

Modules should provide functionality - not take control of your script’s shutdown behavior.

Why Would Someone Put an END Block in a Perl Module?

The first and most obvious answer is that they were unaware of how DESTROY blocks can be employed. If you know something about the author and you’re convinced they know better, then why else?

I theorize that the author was trying to create a module that would encapsulate functionality that he would use in EVERY script he wrote for the application. While the faux pas might be forgiven I’m not ready to put my wagging finger back in its holster.

If you want to write a wrapper for all your scripts and you’ve already settled on using a Perl module to do so, then please for all that is good and holy do it right. Here are some potential guidelines for that wrapper:

• A new() method that instantiates your wrapper
• An init() method that encapsulates the common startup operations with options to control whether some are executed or not
• A run() method that executes the functionality for the script
• A finalize() method for executing cleanup procedures
• POD that describes the common functionality provided as well as any options for controlling their invocation

All of these methods could be overridden if you just use a plain ‘ol Perl module. For those that prefer composition over inheritance, use a role with something like Role::Tiny to provide those universally required methods. Using Role::Tiny provides better flexibility by allowing you to use those methods before or after your modifications to their behavior.

How Can We Take Back Control?

The particular script I was working on included a module(whose name I shall not speak) that included such an END block. My script should have exited cleanly and quietly. Instead, it produced mysterious messages during shutdown. Worse yet I feared some undocumented behaviors and black magic might have been conjured up during that process! After a bit of debugging, I found the culprit:

• The module had an END block baked into it
• This END block printed debug messages to STDERR while doing other cleanup operations
• Worse, it ran unconditionally when my script terminated

The Naive Approach

My initial attempt to suppress the module’s END block:

END {
    use POSIX;
    POSIX::_exit(0);
}

This works as long as my script exits normally. But if my script dies due to an error, the rogue END block still executes. Not exactly the behavior I want.

A Better Approach

Here’s what I want to happen:

• Prevent any rogue END block from executing.
• Handle errors gracefully.

• Ensure my script always exits with a meaningful status code.

• A better method for scripts that need an END block to claw back control:

use English qw(-no_match_vars);
use POSIX ();

my $retval = eval {
    return main();
};

END {
  if ($EVAL_ERROR) {
      warn "$EVAL_ERROR";  # Preserve exact error message
  }

  POSIX::_exit( $retval // 1 );
}

• Bypasses all END blocks – Since POSIX::_exit() terminates the process immediately
• Handles errors cleanly – If main() throws an exception, we log it without modifying the message
• Forces explicit return values – If main() forgets to return a status, we default to 1, ensuring no silent failures.
• Future maintainers will see exactly what’s happening

Caveat Emptor

Of course, you should know what behavior you are bypassing if you decide to wrestle control back from some misbehaving module. In my case, I knew that the behaviors being executed in the END block could safely be ignored. Even if they couldn’t be ignored, I can still provide those behaviors in my own cleanup procedures.

Isn’t this what future me or the poor wretch tasked with a dumpster dive into a legacy application would want? Explicitly seeing the whole shebang without hours of scratching your head looking for mysterious messages that emanate from the depths is priceless. It’s gold, Jerry! Gold!

Next up…a better wrapper.

In our previous posts, we explored how Apache 2.4 changed its handling of directory requests when DirectorySlash Off is set, breaking the implicit /dir → /dir/ redirect behavior that worked in Apache 2.2. We concluded that while an external redirect is the only reliable fix, this change in behavior led us to an even bigger question:

Is this a bug or an intentional design change in Apache?

After digging deeper, we’ve uncovered something critically important that is not well-documented:

Apache does not restart the request cycle after an internal rewrite, and this can break expected behaviors like DirectoryIndex.

This post explores why this happens, whether it’s a feature or a bug, and why Apache’s documentation should explicitly clarify this behavior.

What Happens When Apache Internally Rewrites a Request?

Let’s revisit the problem: we tried using an internal rewrite to append a trailing slash for directory requests:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ /$1/ [L]

Expected Behavior: - Apache should internally rewrite /setup to /setup/. - Since DirectoryIndex index.roc is set, Apache should serve index.roc.

Actual Behavior: - Apache internally rewrites /setup to /setup/, but then immediately fails with a 403 Forbidden. - The error log states: AH01276: Cannot serve directory /var/www/vhosts/treasurersbriefcase/htdocs/setup/: No matching DirectoryIndex (none) found - Apache is treating /setup/ as an empty directory instead of recognizing index.roc.

The Key Issue: Apache Does Not Restart Request Processing After an Internal Rewrite

Unlike what many admins assume, Apache does not start over after an internal rewrite.

How Apache Processes Requests (Simplified)

1. The request arrives (/setup).
2. Apache processes mod_rewrite.
3. Apache determines how to serve the request.
4. If an index file (index.html, index.php, etc.) exists, DirectoryIndex resolves it.

Why Internal Rewrites Don’t Restart Processing

• Apache processes mod_rewrite before it checks DirectoryIndex.
• Once a rewrite occurs, Apache continues processing from where it left off.
• This means it does not re-check DirectoryIndex after an internal rewrite.
• Instead, it sees /setup/ as an empty directory with no default file and denies access with 403 Forbidden.

Is This a Bug or a Feature?

First, let’s discuss why this worked in Apache 2.2.

The key reason internal rewrites worked in Apache 2.2 is that Apache restarted the request processing cycle after a rewrite. This meant that:

• After an internal rewrite, Apache treated the rewritten request as a brand-new request.
• As a result, it re-evaluated DirectoryIndex and correctly served index.html, index.php, or any configured default file.
• Since DirectorySlash was handled earlier in the request cycle, Apache 2.2 still applied directory handling rules properly, even after an internal rewrite.

In Apache 2.4, this behavior changed. Instead of restarting the request cycle, Apache continues processing the request from where it left off. This means that after an internal rewrite, DirectoryIndex is never reprocessed, leading to the 403 Forbidden errors we encountered. This fundamental change explains why no internal solution works the way it did in Apache 2.2.

Why It’s Likely an Intentional Feature

• In Apache 2.2, some rewrites did restart the request cycle, which was seen as inefficient.
• In Apache 2.4, request processing was optimized for performance, meaning it does not restart after an internal rewrite.
• The behavior is consistent across different Apache 2.4 installations.
• Some discussions in Apache’s mailing lists and bug tracker mention this as “expected behavior.”

Why This is Still a Problem

• This behavior is not explicitly documented in mod_rewrite or DirectoryIndex docs.
• Most admins expect Apache to reprocess the request fully after a rewrite.
• The lack of clarity leads to confusion and wasted debugging time.

Implications for Apache 2.4 Users

1. Mod_Rewrite Behavior is Different From What Many Assume

• Internal rewrites do not restart request processing.
• DirectoryIndex is only evaluated once, before the rewrite happens.
• This is not obvious from Apache’s documentation.

2. The Only Reliable Fix is an External Redirect

Since Apache won’t reprocess DirectoryIndex, the only way to guarantee correct behavior is to force a new request via an external redirect:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]

This forces Apache to start a completely new request cycle, ensuring that DirectoryIndex is evaluated properly.

3. Apache Should Improve Its Documentation

We believe this behavior should be explicitly documented in: - mod_rewrite documentation (stating that rewrites do not restart request processing). - DirectoryIndex documentation (noting that it will not be re-evaluated after an internal rewrite).

This would prevent confusion and help developers troubleshoot these issues more efficiently.

Conclusion: A Feature, But a Poorly Documented One

• The fact that Apache does not restart processing after an internal rewrite is likely an intentional design choice.
• However, this is not well-documented, leading to confusion.
• The only solution remains an external redirect to force a fresh request cycle.
• We believe Apache should update its documentation to reflect this behavior more clearly.

In our previous blog post, we detailed a clever external redirect solution to address the Apache 2.4 regression that broke automatic directory handling when DirectorySlash Off was set. We teased the question: Can we achieve the same behavior with an internal redirect? Spoiler alert - Nope.

In this follow-up post, we’ll explore why internal rewrites fall short, our failed attempts to make them work, and why the external redirect remains the best and only solution.

The Apache 2.2 vs. 2.4 Behavior Shift

How Apache 2.2 Handled Directory Requests

• If a user requested /dir without a trailing slash, Apache would automatically redirect them to /dir/.
• Apache would then serve the directory’s index.html (or any file specified by DirectoryIndex).
• This behavior was automatic and required no additional configuration.

What Changed in Apache 2.4

• When DirectorySlash Off is set, Apache stops auto-redirecting directories.
• Instead of treating /dir as /dir/, Apache tries to serve /dir as a file, which leads to 403 Forbidden errors.
• DirectoryIndex no longer inherits globally from the document root - each directory must be explicitly configured to serve an index file.
• Apache does not reprocess DirectoryIndex after an internal rewrite.

The Internal Rewrite Attempt

Our Initial Idea

Since Apache wasn’t redirecting directories automatically anymore, we thought we could internally rewrite requests like this:

 RewriteEngine On

 # If the request is missing a trailing slash and is a directory, rewrite it internally
 RewriteCond %{REQUEST_URI} !/$
 RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
 RewriteRule ^(.*)$ /$1/ [L]

Why This Doesn’t Work:

• While this internally rewrites the request, Apache does not reprocess DirectoryIndex after the rewrite.
• If DirectoryIndex is not explicitly defined for each directory, Apache still refuses to serve the file and throws a 403 Forbidden.
• Unlike Apache 2.2, Apache 2.4 treats /dir as a raw file request instead of checking for an index page.

The Per-Directory Fix (Which Also Fails)

To make it work, we tried to manually configure every directory:

<Directory "/var/www/vhosts/treasurersbriefcase/htdocs/setup/">
    Require all granted
    DirectoryIndex index.roc
</Directory>

Why This Also Fails:

• Apache does not reprocess DirectoryIndex after an internal rewrite. Even though DirectoryIndex index.roc is explicitly set, Apache never reaches this directive after rewriting /setup to /setup/.
• Apache still treats /setup/ as an empty directory, leading to a 403 Forbidden error.
• The only way to make this work per directory would be to use an external redirect to force a new request cycle.

This means that even if we were willing to configure every directory manually, it still wouldn’t work as expected.

FallbackResource (Why This Was a Dead End)

We briefly considered whether FallbackResource could help by redirecting requests for directories to their respective index files:

<Directory "/var/www/vhosts/treasurersbriefcase/htdocs/setup/">
    DirectoryIndex index.roc
    Options -Indexes
    Require all granted
    FallbackResource /setup/index.roc
</Directory>

Why This Makes No Sense

• FallbackResource is designed to handle 404 Not Found errors, not 403 Forbidden errors.
• Since Apache already recognizes /setup/ as a valid directory but refuses to serve it, FallbackResource is never triggered.
• This does not address the fundamental issue that Apache does not reprocess DirectoryIndex after an internal rewrite.

This was a red herring in our troubleshooting FallbackResource was never a viable solution.

Sledge Hammer Approach: Direct Rewrite to the Index File

Another way to handle this issue would be to explicitly rewrite directory requests to their corresponding index file, bypassing Apache’s DirectoryIndex handling entirely:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ /$1/index.roc [L]

It works…

• It completely avoids Apache’s broken DirectoryIndex handling in Apache 2.4.
• No need for DirectorySlash Off, since the request is rewritten directly to a file.
• It prevents the 403 Forbidden issue because Apache is no longer serving a bare directory.

…but it’s not ideal

• Every directory would need an explicit rewrite rule to its corresponding index file.
• If different directories use different index files (index.html, index.php, etc.), additional rules would be required.
• This does not scale well without complex conditional logic.

While this approach technically works, it reinforces our main conclusion: - Apache 2.4 no longer restarts the request cycle after a rewrite, so we need to account for it manually. - The external redirect remains the only scalable solution.

Why the External Redirect is the Best Approach

Instead of fighting Apache’s new behavior, we can work with it using an external redirect:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]

Why This Works Perfectly

• Redirects /dir to /dir/ before Apache even tries to serve it.
• Preserves DirectoryIndex behavior globally, just like Apache 2.2.
• No need for per-directory configuration.
• Ensures SEO-friendly canonical URLs with the correct trailing slash.

Conclusion

So, can we solve with an internal redirect?

• NO! … because Apache does not reprocess DirectoryIndex after an internal rewrite.
• Even explicit per-directory DirectoryIndex settings fail if Apache has already decided the request is invalid.
• FallbackResource never applied, since Apache rejected the request with 403 Forbidden, not 404 Not Found.

Does our external redirect solution still hold up?

Yes!!, and in fact, it’s not just the best solution - it’s the only reliable one.

Lessons Learned

• Apache 2.4 fundamentally changed how it handles directory requests when DirectorySlash Off is set.
• Internal rewrites cannot fully restore Apache 2.2 behavior because Apache does not restart request processing after a rewrite.
• The only way to ensure correct behavior is to use an external redirect before Apache attempts to serve the request.

If you haven’t read our original post yet, check it out here for the full explanation of our external redirect fix.

In part III of this series we’ll beat this dead horse and potentially explain why this was a problem in the first place…

Introduction

I’m currently re-architecting my non-profit accounting SaaS (Treasurer’s Briefcase) to use Docker containers for a portable development environment and eventually for running under Kubernetes. The current architecture designed in 2012 uses an EC2 based environment for both development and run-time execution.

As part of that effort I’m migrating from Apache 2.2 to 2.4. In the new development environment I’m using my Chromebook to access the containerized application running on the EC2 dev server. I described that setup in my previous blog. In that setup I’m accessing the application on port 8080 as http://local-dev:8080.

If, as in the setup describe in my blog, you are running Apache on a non-standard port (e.g., :8080) - perhaps in Docker, EC2, or via an SSH tunnel you may have noticed an annoying issue after migrating from Apache 2.2 to Apache 2.4…

Apache 2.4 Drops the Port When Redirecting Directories!

Previously, when requesting a directory without a trailing slash (e.g., /setup), Apache automatically redirected to /setup/ while preserving the port. However, in Apache 2.4, the redirect is done externally AND drops the port, breaking relative URLs and form submissions.

For example let’s suppose you have a form under the /setup directory that has as its action “next-step.html”. The expected behavior on that page would be to post to the page /setup/next-step.html. But what really happens is different. You can’t even get to the form in the first place with the URL http://local-dev:8080/setup!

• Expected Redirect (Apache 2.2):\ http://yourserver:8080/setup => http://yourserver:8080/setup/
• Actual Redirect (Apache 2.4, Broken):\ http://yourserver:8080/setup => http://yourserver/setup/ (port 8080 is missing!)

This causes problems for some pages in web applications running behind Docker, SSH tunnels, and EC2 environments, where port forwarding is typically used.

Investigating the Issue

If you’re experiencing this problem, you can confirm it by running:

curl -IL http://yourserver:8080/setup

You’ll likely see:

HTTP/1.1 301 Moved Permanently
Location: http://yourserver/setup/

Apache dropped 8080 from the redirect, causing requests to break.

Workarounds

Several workarounds exist, but they don’t work in our example.

• Disabling DirectorySlash: Prevents redirects but causes 403 Forbidden errors when accessing directories.
• Using FallbackResource: Works, but misroutes unrelated requests.
• Hardcoding the port in rewrite rules: Not flexible across different environments.

Instead, we need a solution that dynamically preserves the port when necessary.

The Fix

To restore Apache 2.2 behavior, we can use a rewrite rule that only preserves the port if it was in the original request.

Apache 2.4 Fix: Port-Preserving Rewrite Rule

<VirtualHost *:8080>
    ServerName yourserver
    DocumentRoot /var/www/html

    <Directory /var/www/html>
        Options -Indexes +FollowSymLinks
        DirectoryIndex index.html index.php
        Require all granted
        DirectorySlash On  # Keep normal Apache directory behavior
    </Directory>

    # Fix Apache 2.4 Directory Redirects: Preserve Non-Standard Ports
    RewriteEngine On
    RewriteCond %{REQUEST_URI} !/$
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
    RewriteCond %{SERVER_PORT} !^80$ [OR]
    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]

    UseCanonicalName Off
</VirtualHost>

Explanation

• Automatically appends a missing trailing slash (/setup => /setup/).
• Preserves the port only if it’s non-standard (!=80, !=443)
• Avoids hardcoding :8080, making it flexible for any non-standard port
• Restores Apache 2.2 behavior while keeping things modern and correct.

Example: Running Apache in Docker on EC2 via SSH Tunnel

The Setup (see previous blog)

1. Docker container running Apache on port 80 inside an EC2 instance.
2. Firewall rules allow only my home IP to access the server.
3. SSH tunnel (jump box) forwards port 80 securely.
4. Chromebook’s SSH settings forward port 8080 locally to 80 on the jump box.

How the Fix Helps

• Previously, /setup redirected externally without the port, causing failures.
• This fix use mod_rewrite and a RewriteRule that ensures that port 8080 is preserved

Conclusion

Apache 2.4’s port-dropping behavior is an unexpected regression from 2.2, but we can fix it with a simple rewrite rule that restores the expected behavior without breaking anything.

If you’re running Docker, EC2, or SSH tunnels, this is a fix that will prevent you from jumping through hoops by altering the application or changing your networking setup.

Postamble

Hmmmm…maybe we can use internal redirects instead of an external redirect??? Stay tuned.

Introduction

If you’re doing some development on a remote server that you access via a bastion host (e.g., an EC2 instance), and you want a seamless way to work from a Chromebook, you can set up an SSH tunnel through the bastion host to access your development server.

This guide outlines how to configure a secure and efficient development workflow using Docker, SSH tunnels, and a Chromebook.

Motivation

Your Chromebook is a great development environment, but truth be told, the cloud is better. Why? Because you can leverage a bucket load of functionality, resources and infrastructure that is powerful yet inexpensive. Did I mention backups? My Chromebook running a version of debian rocks, but in general I use it as a conduit to the cloud.

So here’s the best of both worlds. I can use a kick-butt terminal (terminator) on my Chromie and use its networking mojo to access my web servers running in the cloud.

Network Setup Overview

In this setup:

• The Chromebook runs Linux and connects via SSH to the bastion host.
• The bastion host acts as an intermediary, forwarding requests to the private EC2 development server.
• The EC2 instance is firewalled and only accessible from the bastion host.
• Port 80 on the EC2 instance is mapped to port 8080 locally on the Chromebook through the SSH tunnel. You’ll need to set that up on your Chromebook in the Linux development environment settings.

Network Diagram

Here’s what it looks like in ASCII art…

   +--------------+    +--------------+    +--------------+
   | Chromebook   |    | Bastion Host |    |     EC2      |
   |              | 22 |              | 22 |              |
   |  Local SSH   |----|   Jump Box   |----| Development  |
   |  Tunnel:8080 | 80 | (Accessible) | 80 |  Server      |
   +--------------+    +--------------+    +--------------+

Setting Up the SSH Tunnel

To create an SSH tunnel through the bastion host:

ssh -N -L 8080:EC2_PRIVATE_IP:80 user@bastion-host

Explanation:

• -N: Do not execute remote commands, just forward ports.
• -L 8080:EC2_PRIVATE_IP:80: Forwards local port 8080 to port 80 on the development server (EC2 instance).
• user@bastion-host: SSH into the bastion host as user.

Once connected, any request to localhost:8080 on the Chromebook will be forwarded to port 80 on the EC2 instance.

Making It Persistent on Your Chromebook

To maintain the tunnel connection automatically:

1. Use an SSH config file (~/.ssh/config):

Host bastion
    HostName bastion-host
    User your-user
    IdentityFile ~/.ssh/id_rsa
    LocalForward 8080 EC2_PRIVATE_IP:80
    ServerAliveInterval 60
    ServerAliveCountMax 3

1. Start the tunnel in the background:

ssh -fN bastion

1. Verify it is working:

curl -I http://localhost:8080

You should see a response from your EC2 instance’s web server.

Integrating with Docker on EC2

If your EC2 instance runs a Dockerized web application, expose port 80 from the container:

docker run -d -p 80:80 my-web-app

Now, accessing http://localhost:8080 on your Chromebook browser will open the web app running inside the Docker container on EC2.

Final Thoughts

This setup allows you to securely access a remote development environment from a Chromebook, leveraging SSH tunneling through a bastion host.

• Why This Works:
  • Keeps the EC2 instance private while still making it accessible for development.
  • Allows seamless local access (localhost:8080) to a remote web app.
  • Works even when using strict firewall rules.

Now you can develop on remote servers with a Chromebook, as if they were local!

This effort is the terminal event (I hope) in my adventure in Perl packaging that led me down the rabbit hole of CloudFront distributions?

My adventure in packaging started many years ago when it seemed like a good idea to use RPMs. No really, it was!

The distribtions embraced Perl and life was good…until of course it wasn’t. Slowly but surely, those modules we wanted weren’t available in any of the repos. Well, here we are in 2025 and you can’t even grovel enough to get AWS to include Perl in its images? Sheesh, really? I have to yum install perl-core? Seems like someone has specifically put Perl on the shit list.

I’ve been swimming upstream for years using cpanspec and learning how to create my own yum repos with all of the stuff Amazon just didn’t want to package. I’ve finally given up. CPAN forever! cpanm is awesome! Okay, yeah, but I’m still not all in with carton. Maybe some day?

Key Features of OrePAN2::S3

• Seamless Integration with AWS: OrePAN2::S3 leverages Amazon S3 for storage and CloudFront for content delivery, providing a highly available, scalable solution for hosting your private CPAN repository.

• User-Friendly Setup: The distribution offers straightforward scripts and configurations, enabling you to set up your DarkPAN with minimal effort…I hope. If you find some points of friction, log an issue.

• Flexible Deployment Options: Whether you prefer a simple S3-backed website or a full-fledged CloudFront distribution, OrePAN2::S3 accommodates both setups to suit your specific needs. If you really just want a website enable S3 bucket to serve as your DarkPAN we got your back. But be careful…

Getting Started:

To begin using OrePAN2::S3, ensure you have the following prerequisites:

1. AWS Account: An active Amazon Web Services account.

2. S3 Bucket: A designated S3 bucket to store your CPAN modules.

3. CloudFront Distribution (optional): For enhanced content delivery and caching.

Detailed instructions for setting up your S3 bucket and CloudFront distribution are available in the project’s repository.

Why Choose OrePAN2::S3?

Managing a private CPAN repository can be complex, but with OrePAN2::S3, the process becomes efficient and scalable. By harnessing the power of AWS services, this distribution ensures your Perl modules are readily accessible and securely stored.

Oh, and let’s give credit where credit is due. This is all based on OrePAN2.

For more information and to access the repository, visit: https://github.com/rlauer6/OrePAN2-S3

We look forward to your feedback and contributions to make OrePAN2::S3 even more robust and user-friendly.