Creating AWS Resources…let me count the ways

You need to create an S3 bucket, an SQS queue, an IAM policy and a few other AWS resources. But how?…TIMTOWTDI

The Console

• Pros: visual, immediate feedback, no tooling required, great for exploration
• Cons: not repeatable, not version controllable, opaque, clickops doesn’t scale, “I swear I configured it the same way”

The AWS CLI

• Pros: scriptable, composable, already installed, good for one-offs
• Cons: not idempotent by default, no state management, error handling is manual, scripts can grow into monsters

CloudFormation

• Pros: native AWS, state managed by AWS, rollback support, drift detection
• Cons: YAML/JSON verbosity, slow feedback loop, stack update failures are painful, error messages are famously cryptic, proprietary to AWS, subject to change without notice

Terraform

• Pros: multi-cloud, huge community, mature ecosystem, state management, plan before apply
• Cons: state file complexity, backend configuration, provider versioning, HCL is yet another language to learn, overkill for small projects, often requires tricks & contortions

Pulumi

• Pros: real programming languages, familiar abstractions, state management
• Cons: even more complex than Terraform, another runtime to install and maintain

CDK

• Pros: real programming languages, generates CloudFormation, good for large organizations
• Cons: CloudFormation underneath means CloudFormation problems, Node.js dependency

…and the rest of crew…

Ansible, AWS SAM, Serverless Framework - each with their own opinions, dependencies, and learning curves.

Every option beyond the CLI adds a layer of abstraction, a new language or DSL, a state management story, and a new thing to learn and maintain. For large teams managing hundreds of resources across multiple environments that overhead is justified. For a solo developer or small team managing a focused set of resources it can feel like overkill.

Even in large organizations, not every project should be conflated into the corporate infrastructure IaC tool. Moreover, not every project gets the attention of the DevOps team necessary to create or support the application infrastructure.

What if you could get idempotent, repeatable, version-controlled infrastructure management using tools you already have? No new language, no state backend, no provider versioning. Just make, bash, a scripting language you’re comfortable with, and your cloud provider’s CLI.

And yes…my love affair with make is endless.

We’ll use AWS examples throughout, but the patterns apply equally to Google Cloud (gcloud) and Microsoft Azure (az). The CLI tools differ, the patterns don’t.

A word about the AWS CLI --query option

Before you reach for jq, perl, or python to parse CLI output, it’s worth knowing that most cloud CLIs have built-in query support. The AWS CLI’s --query flag implements JMESPath - a query language for JSON that handles the majority of filtering and extraction tasks without any additional tools:

# get a specific field
aws lambda get-function \
    --function-name my-function \
    --query 'Configuration.FunctionArn' \
    --output text

# filter a list
aws sqs list-queues \
    --query 'QueueUrls[?contains(@, `my-queue`)]|[0]' \
    --output text

--query is faster, requires no additional dependencies, and keeps your pipeline simple. Reach for it first. When it falls short - complex transformations, arithmetic, multi-value extraction - that’s when a one-liner earns its place:

# perl
aws lambda get-function --function-name my-function | \
    perl -MJSON -n0 -e '$l=decode_json($_); print $l->{Configuration}{FunctionArn}'

# python
aws lambda get-function --function-name my-function | \
    python3 -c "import json,sys; d=json.load(sys.stdin); print(d['Configuration']['FunctionArn'])"

Both get the job done. Use whichever lives in your shed.

What is Idempotency?

The word comes from mathematics - an operation is idempotent if applying it multiple times produces the same result as applying it once. Sort of like those ID10T errors…no matter how hard or how many times that user clicks on that button they get the same result.

In the context of infrastructure management it means this: running your resource creation script twice should have exactly the same outcome as running it once. The first run creates the resource. The second run detects it already exists and does nothing - no errors, no duplicates, no side effects.

This sounds simple but it’s surprisingly easy to get wrong. A naive script that just calls aws lambda create-function will fail on the second run with a ResourceConflictException. A slightly better script wraps that in error handling. A truly idempotent script never attempts to create a resource it knows already exists.

And it works in both directions. The idempotent bug - running a failing process repeatedly and getting the same error every time - is what happens when your failure path is idempotent too. Consistently wrong, no matter how many times you try. The patterns we’ll show are designed to ensure that success is idempotent while failure always leaves the door open for the next attempt.

Cloud APIs fall into four distinct behavioral categories when it comes to idempotency, and your tooling needs to handle each one differently:

Case 1 - The API is idempotent and produces output

Some APIs can be called repeatedly without error and return useful output each time - whether the resource was just created or already existed. aws events put-rule is a good example - it returns the rule ARN whether the rule was just created or already existed. The pattern: call the read API first, capture the output, call the write API only if the read returned nothing.

Case 2 - The API is idempotent but produces no output

Some write APIs succeed silently - they return nothing on success. aws s3api put-bucket-notification-configuration is a good example. It will happily overwrite an existing configuration without complaint, but returns no output to confirm success. The pattern: call the API, synthesize a value for your sentinel using && echo to capture something meaningful on success.

Case 3 - The API is not idempotent

Some APIs will fail with an error if you try to create a resource that already exists. aws lambda add-permission returns ResourceConflictException if the statement ID already exists. aws lambda create-function returns ResourceConflictException if the function already exists. These APIs give you no choice - you must query first and only call the write API if the resource is missing.

Case 4 - The API call fails

Any of the above can fail - network errors, permission problems, invalid parameters. When a call fails you must not leave behind a sentinel file that signals success. A stale sentinel is worse than no sentinel - it tells Make the resource exists when it doesn’t, and subsequent runs silently skip the creation step. The patterns: || rm -f $@ when writing directly, or else rm -f $@ when capturing to a variable first.

The Sentinel File

Before we look at the four patterns in detail, we need to introduce a concept that ties everything together: the sentinel file.

A sentinel file is simply a file whose existence signals that a task has been completed successfully. It contains no magic - it might hold the output of the API call that created the resource, or it might just be an empty file created with touch. What matters is that it exists when the task succeeded and doesn’t exist when it hasn’t.

make has used this pattern since the 1970s. When you declare a target in a Makefile, make checks whether a file with that name exists before deciding whether to run the recipe. If the file exists and is newer than its dependencies, make skips the recipe entirely. If the file doesn’t exist, make runs the recipe to create it.

For infrastructure management this is exactly the behavior we want:

my-resource:
    @value="$$(aws some-service describe-resource \
            --name $(RESOURCE_NAME) 2>&1)"; \
    if [[ -z "$$value" || "$$value" = "ResourceNotFound" ]]; then \
        value="$$(aws some-service create-resource \
            --name $(RESOURCE_NAME))"; \
    fi; \
    test -e $@ || echo "$$value" > $@

The first time you run make my-resource the file doesn’t exist, the recipe runs, the resource is created, and the API response is written to the sentinel file my-resource. The second time you run it, make sees the file exists and skips the recipe entirely - zero API calls.

When an API call fails we want to be sure we do not create the sentinel file. We’ll cover the failure case in more detail in Pattern 4 of the next section.

The Four Patterns

Armed with the sentinel file concept and an understanding of the four API behavioral categories, let’s look at concrete implementations of each pattern.

Pattern 1 - Idempotent API with output

The simplest case. Query the resource first - if it exists capture the output and write the sentinel. If it doesn’t exist, create it, capture the output, and write the sentinel. Either way you end up with a sentinel containing meaningful content.

The SQS queue creation is a good example:

sqs-queue:
    @queue="$$(aws sqs list-queues \
        --query 'QueueUrls[?contains(@, `$(QUEUE_NAME)`)]|[0]' \
        --output text --profile $(AWS_PROFILE) 2>&1)"; \
    if echo "$$queue" | grep -q 'error\|Error'; then \
        echo "ERROR: list-queues failed: $$queue" >&2; \
        exit 1; \
    elif [[ -z "$$queue" || "$$queue" = "None" ]]; then \
        queue="$(QUEUE_NAME)"; \
        aws sqs create-queue --queue-name $(QUEUE_NAME) \
            --profile $(AWS_PROFILE); \
    fi; \
    test -e $@ || echo "$$queue" > $@

Notice --query doing the filtering work before the output reaches the shell. No jq, no pipeline - the AWS CLI extracts exactly what we need. The result is either a queue URL or empty. If empty we create. Either way $$queue ends up with a value and the sentinel is written exactly once.

The EventBridge rule follows the same pattern:

lambda-eventbridge-rule:
    @rule="$$(aws events describe-rule \
            --name $(RULE_NAME) \
            --profile $(AWS_PROFILE) 2>&1)"; \
    if echo "$$rule" | grep -q 'ResourceNotFoundException'; then \
        rule="$$(aws events put-rule \
            --name $(RULE_NAME) \
            --schedule-expression "$(SCHEDULE_EXPRESSION)" \
            --state ENABLED \
            --profile $(AWS_PROFILE))"; \
    elif echo "$$rule" | grep -q 'error\|Error'; then \
        echo "ERROR: describe-rule failed: $$rule" >&2; \
        exit 1; \
    fi; \
    test -e $@ || echo "$$rule" > $@

Same shape - query, create if missing, write sentinel once.

Pattern 2 - Idempotent API with no output

Some APIs succeed silently. aws s3api put-bucket-notification-configuration is the canonical example - it happily overwrites an existing configuration and returns nothing. No output means nothing to write to the sentinel.

The solution is to synthesize a value using &&:

define notification_configuration =
use JSON;

my $lambda_function = $ENV{lambda_function};
my $function_arn = decode_json($lambda_function)->{Configuration}->{FunctionArn};

my $configuration = {
 LambdaFunctionConfigurations => [ {
   LambdaFunctionArn => $function_arn,
   Events => [ split ' ', $ENV{s3_event} ],
  }
 ]
};

print encode_json($configuration);
endef

export s_notification_configuration = $(value notification_configuration)

lambda-s3-trigger: lambda-s3-permission
        temp="$$(mktemp)"; trap 'rm -f "$$temp"' EXIT; \
        lambda_function="$$(cat lambda-function)"; \
        echo $$(s3_event="$(S3_EVENT)" lambda_function="$$lambda_function" \
          perl -e "$$s_notification_configuration") > $$temp; \
        trigger="$$(aws s3api put-bucket-notification-configuration \
            --bucket $(BUCKET_NAME) \
            --notification-configuration file://$$temp \
            --profile $(AWS_PROFILE) && cat $$temp)"; \
        test -e $@ || echo "$$trigger" > $@

The && cat $$temp is the key. If the API call succeeds the && fires and $$trigger gets the configuration JSON string - something meaningful to write to the sentinel. If the API call fails && doesn’t fire, $$trigger stays empty because the Makefile recipe aborts.

Using a scriptlet (s_notification_configuration) might seem like overkill, but it’s worth not having to fight shell quoting issues!

Writing JSON used in many AWS API calls to a temporary file is usually a better way than passing a string on the command line. Unless you wrap the JSON in quotes you’ll be fighting shell quoting and interpolation issues…and of course you can write your scriptlets in Perl or Python!

Pattern 3 - Non-idempotent API

Some APIs are not idempotent - they fail with a ResourceConflictException or similar if the resource already exists. aws lambda add-permission and aws lambda create-function are both in this category. There is no “create or update” variant - you must check existence first and only call the write API if the resource is missing.

The Lambda S3 permission target is a good example:

lambda-s3-permission: lambda-function s3-bucket
        @permission="$$(aws lambda get-policy \
                --function-name $(FUNCTION_NAME) \
                --profile $(AWS_PROFILE) 2>&1)"; \
        if echo "$$permission" | grep -q 'ResourceNotFoundException' || \
           ! echo "$$permission" | grep -q s3.amazonaws.com; then \
            permission="$$(aws lambda add-permission \
                --function-name $(FUNCTION_NAME) \
                --statement-id s3-trigger-$(BUCKET_NAME) \
                --action lambda:InvokeFunction \
                --principal s3.amazonaws.com \
                --source-arn arn:aws:s3:::$(BUCKET_NAME) \
                --profile $(AWS_PROFILE))"; \
        elif echo "$$permission" | grep -q 'error\|Error'; then \
            echo "ERROR: get-policy failed: $$permission" >&2; \
            exit 1; \
        fi; \
        if [[ -n "$$permission" ]]; then \
            test -e $@ || echo "$$permission" > $@; \
        else \
            rm -f $@; \
        fi

A few things worth noting here…

• get-policy returns the full policy document which may contain multiple statements - we check for the presence of s3.amazonaws.com specifically using ! grep -q rather than just checking for an empty response. This handles the case where a policy exists but doesn’t yet have the S3 permission we need.
• The sentinel is only written if $$permission is non-empty after the if block. This covers the case where get-policy returns nothing and add-permission also fails - the sentinel stays absent and the next make run will try again.
• We pipe errors to our bash variable to detect the case where the resource does not exist or there may have been some other error. When other failures are possible 2>&1 combined with specific error string matching gives you both idempotency and visibility. Swallowing errors silently (2>/dev/null) is how idempotent bugs are born.

Pattern 4 - Failure handling

This isn’t a separate pattern so much as a discipline that applies to all three of the above. There are two mechanisms depending on how the sentinel is written.

Case 1: When the sentinel is written directly by the command:

aws lambda create-function ... > $@ || rm -f $@

|| rm -f $@ ensures that if the command fails the partial or empty sentinel is immediately cleaned up. Without it make sees the file on the next run and silently skips the recipe - an idempotent bug.

Case 2: When the sentinel is written by capturing output to a variable first:

if [[ -n "$$value" ]]; then \
    test -e $@ || echo "$$value" > $@; \
else \
    rm -f $@; \
fi

The else rm -f $@ serves the same purpose. If the variable is empty - because the API call failed - the sentinel is removed. If the sentinel doesn’t exist yet nothing is written. Either way the next make run will try again.

In both cases the goal is the same: a sentinel file should only exist when the underlying resource exists. A stale sentinel is worse than no sentinel.

Depending on the way your recipe is written you may not need to test the variable that capture the output at all. In Makefiles we .SHELLFLAGS := -ec which causes make to exit immediately if any command in a recipe fails. This means targets that don’t write to $@ - like our sqs-queue target above - don’t need explicit failure handling. make will die loudly and the sentinel won’t be written. In that case you don’t even need to test $$value and can simplify writing of the sentinel file like this:

test -e $@ || echo "$$value" > $@

Conclusion

Creating AWS resources can be done using several different tools…all of them eventually call AWS APIs and process the return payloads. Each of these tools has its place. Each adds something. Each also has a complexity, dependencies, and a learning curve score.

For a small project or a focused set of resources - the kind a solo developer or small team manages for a specific application - you don’t need tools with a high cognitive or resource load. You can use those tools you already have on your belt; make,bash, [insert favorite scripting language here], and aws. And you can leverage those same tools equally well with gcloud or az.

The four patterns we’ve covered handle every AWS API behavior you’ll encounter:

• Query first, create only if missing, write a sentinel
• Synthesize output when the API has none
• Always check before calling a non-idempotent API
• Clean up on failure with || rm -f $@

These aren’t new tricks - they’re straightforward applications of tools that have been around for decades. make has been managing file-based dependencies since 1976. The sentinel file pattern predates cloud computing entirely. We’re just applying them to a new problem.

One final thought. The idempotent bug - running a failing process repeatedly and getting the same error every time - is the mirror image of what we’ve built here. Our goal is idempotent success: run it once, it works. Run it again, it still works. Run it a hundred times, nothing changes. || rm -f $@ is what separates idempotent success from idempotent failure - it ensures that a bad run always leaves the door open for the next attempt rather than cementing the failure in place with a stale sentinel.

Your shed is already well stocked. Sometimes the right tool for the job is the one you’ve had hanging on the wall for thirty years.

Further Reading

• “Advanced Bash-Scripting Guide” - https://tldp.org/LDP/abs/html/index.html
• “GNU Make” - https://www.gnu.org/software/make/manual/html_node/index.html
• Dave Oswald, “Perl One Liners for the Shell” (Perl conference presentation): https://www.slideshare.net/slideshow/perl-oneliners/77841913
• Peteris Krumins, “Perl One-Liners” (No Starch Press): https://nostarch.com/perloneliners
• Sundeep Agarwal, “Perl One-Liners Guide” (free online): https://learnbyexample.github.io/learn_perl_oneliners/
• AWS CLI JMESPath query documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

Even if you’re skeptical about AI writing your code, you’re leaving time on the table.

Many developers have been slow to adopt AI in their workflows, and that’s understandable. As AI coding assistants become more capable the anxiety is real - nobody wants to feel like they’re training their replacement. But we’re not there yet. Skilled developers who understand logic, mathematics, business needs and user experience will be essential to guide application development for the foreseeable future.

The smarter play is to let AI handle the parts of the job you never liked anyway - the documentation, the release notes, the boilerplate tests - while you stay focused on the work that actually requires your experience and judgment. You don’t need to go all in on day one. Here are six places to start.

1. Unit Test Writing

Writing unit tests is one of those tasks most developers know they should do more of and few enjoy doing. It’s methodical, time-consuming, and the worst time to write them is when the code reviewer asks if they pass.

TDD is a fine theory. In practice, writing tests before you’ve vetted your design means rewriting your tests every time the design evolves - which is often. Most experienced developers write tests after the design has settled, and that’s a perfectly reasonable approach.

The important thing is that they get written at all. Even a test that simply validates use_ok(qw(Foo::Bar)) puts scaffolding in place that can be expanded when new features are added or behavior changes. A placeholder is infinitely more useful than nothing.

This is where AI earns its keep. Feed it a function or a module and it will identify the code paths that need coverage - the happy path, the edge cases, the boundary conditions, the error handling. It will suggest appropriate test data sets including the inputs most likely to expose bugs: empty strings, nulls, negative numbers, off-by-one values - the things a tired developer skips.

You review it, adjust it, own it. AI did the mechanical work of thinking through the permutations. You make sure it reflects how your code is actually used in the real world.

2. Documentation

“Documentation is like sex: when it’s good, it’s very, very good; and when it’s bad, it’s better than nothing.” - said someone somewhere.

Of course, there are developers that justify their disdain for writing documentation with one of two arguments (or both):

1. The code is the documentation
2. Documentation is wrong the moment it is written

It is true, the single source of truth regarding what code actually does is the code itself. What it is supposed to do is what documentation should be all about. When they diverge it’s either a defect in the software or a misunderstanding of the business requirement captured in the documentation.

Code that changes rapidly is difficult to document, but the intent of the code is not. Especially now with AI. It is trivial to ask AI to review the current documentation and align it with the code, negating point #2.

Feed AI a module and ask it to generate POD. It will describe what the code does. Your job is to verify that what it does is what it should do - which is a much faster review than writing from scratch.

3. Release Notes

If you’ve read this far you may have noticed the irony - this post was written by someone who just published a blog post about automating release notes with AI. So consider this section field-tested.

Release notes sit at the intersection of everything developers dislike: writing prose, summarizing work they’ve already mentally moved on from, and doing it with enough clarity that non-developers can understand what changed and why it matters. It’s the last thing standing between you and shipping.

The problem with feeding a git log to AI is that git logs are written for developers in the moment, not for readers after the fact. “Fix the thing” and “WIP” are not useful release note fodder.

The better approach is to give AI real context - a unified diff, a file manifest, and the actual source of the changed files. With those three inputs AI can identify the primary themes of a release, group related changes, and produce structured notes that actually reflect the architecture rather than just the line changes.

A simple make release-notes target can generate all three assets automatically from your last git tag. Upload them, prompt for your preferred format, and you have a first draft in seconds rather than thirty minutes. Here’s how I built it.

You still edit it. You add color, context, and the business rationale that only you know. But the mechanical work of reading every diff and turning it into coherent prose? Delegated.

4. Bug Triage

Debugging can be the most frustrating and the most rewarding experience for a developer. Most developers are predisposed to love a puzzle and there is nothing more puzzling than a race condition or a dangling pointer. Even though books and posters have been written about debugging it is sometimes difficult to know exactly where to start.

Describe the symptoms, share the relevant code, toss your theory at it. AI will validate or repudiate without ego - no colleague awkwardly telling you you’re wrong. It will suggest where to look, what telemetry to add, and before you know it you’re instrumenting the code that should have been instrumented from the start.

AI may not find your bug, but it will be a fantastic bug buddy.

5. Code Review

Since I’ve started using AI I’ve found that one of the most valuable things I can do with it is to give it my first draft of a piece of code. Anything more than a dozen or so lines is fair game.

Don’t waste your time polishing a piece of lava that just spewed from your noggin. There’s probably some gold in there and there’s definitely some ash. That’s ok. You created the framework for a discussion on design and implementation. Before you know it you have settled on a path.

AI’s strength is pattern recognition. It will recognize when your code needs to adopt a different pattern or when you nailed it. Get feedback. Push back. It’s not a one-way conversation. Question the approach, flag the inconsistencies that don’t feel right - your input into that review process is critical in evolving the molten rock into a solid foundation.

6. Legacy Code Deciphering

What defines “Legacy Code?” It’s a great question and hard to answer. And not to get too racy again, but as it has been said of pornography, I can’t exactly define it but I know it when I see it.

Fortunately (and yes I do mean fortunately) I have been involved in maintaining legacy code since the day I started working for a family run business in 1998. The code I maintained there was born literally in the late 70’s and still, to this day generates millions of dollars. You will never learn more about coding than by maintaining legacy code.

These are the major characteristics of legacy code from my experience (in order of visibility):

1. It generates so much money for a company they could not possibly think of it being unavailable.
2. It is monolithic and may in fact consist of modules in multiple languages.
3. It is grown organically over the decades.
4. It is more than 10 years old.
5. The business rules are not documented, opaque and can only be discerned by a careful reading of the software. Product managers and users think they know what the software does, but probably do not have the entire picture.
6. It cannot easily be re-written (by humans) because of #5.
7. It contains as much dead code that is no longer serving any useful purpose as it does useful code.

I once maintained a C program that searched an ISAM database of legal judgments. The code had been ported from a proprietary in-memory binary tree implementation and was likely older than most of the developers reading this post. The business model was straightforward and terrifying - miss a judgment and we indemnify the client. Every change had to be essentially idempotent. You weren’t fixing code, you were performing surgery on a patient who would sue you if the scar was in the wrong place.

I was fortunate - there were no paydays for a client on my watch. But I wish I’d had AI back then. Not to write the code. To help me read it.

Now, where does AI come in? Points 5, 6, and definitely 7.

Throw a jabberwocky of a function at AI and ask it what it does. Not what it should do - what it actually does. The variable names are cryptic, the comments are either missing or lying, and the original author left the company during the Clinton administration. AI doesn’t care. It reads the code without preconception and gives you a plain English explanation of the logic, the assumptions baked in, and the side effects you never knew existed.

That explanation becomes your documentation. Those assumptions become your unit tests. Those side effects become the bug reports you never filed because you didn’t know they were bugs.

Dead code is where AI particularly shines. Show it a module and ask what’s unreachable. Ask what’s duplicated. Ask what hasn’t been touched in a decade but sits there quietly terrifying anyone who considers deleting it. AI will give you a map of the minefield so you can walk through it rather than around it forever.

Along the way AI will flag security vulnerabilities you never knew were there - input validation gaps, unsafe string handling, authentication assumptions that made sense in 1998 and are a liability today. It will also suggest where instrumentation is missing, the logging and telemetry that would have made every debugging session for the last twenty years shorter. You can’t go back and add it to history, but you can add it now before the next incident.

The irony of legacy code is that the skills required to understand it - patience, pattern recognition, the ability to hold an entire system in your head - are exactly the skills AI complements rather than replaces. You still need to understand the business. AI just helps you read the hieroglyphics.

Conclusion

None of the six items on this list require you to hand over the keys. You are still the architect, the decision maker, the person who understands the business and the user. AI is the tireless assistant who handles the parts of the job that drain your energy without advancing your craft.

The developers who thrive in the next decade won’t be the ones who resisted AI the longest. They’ll be the ones who figured out earliest how to delegate the tedious, the mechanical, and the repetitive - and spent the time they saved on the work that actually requires a human.

You don’t have to go all in. Start with a unit test. Paste some legacy code and ask AI to explain it or document it. Think of AI as that senior developer you go to with the tough problems - the one who has seen everything, judges nothing, and is available at 3am when the production system is on fire.

Only this one never sighs when you knock on the door.

You’ve just finished a marathon refactoring - perhaps splitting a monolithic script into proper modules-and now you need to write the release notes. You could feed an AI a messy git log, but if you want high-fidelity summaries that actually understand your architecture, you need to provide better context.

The Solution: AI Loves Boring Tasks

…and is pretty good at them too!

Instead of manually describing changes or hoping it can interpret my ChangeLog, I’ve automated the production of three ephemeral “Sidecar” assets. These are generated on the fly, uploaded to the LLM, and then purged after analysis - no storage required.

The Assets

• The Manifest (.lst): A simple list of every file touched, ensuring the AI knows the exact scope of the release.
• The Logic (.diffs): A unified diff (using git diff --no-ext-diff) that provides the “what” and “why” of every code change.
• The Context (.tar.gz): This is the “secret sauce.” It contains the full source of the changed files, allowing the AI to see the final implementation - not just the delta.

The Makefile Implementation

If you’ve read any of my blog posts you know I’m a huge Makefile fan. To automate this I’m naturally going to add a recipe to my Makefile or Makefile.am.

First, we explicitly set the shell to /usr/bin/env bash to ensure features like brace expansion work consistently across all dev environments.

# Ensure a portable bash environment for advanced shell features
SHELL := /usr/bin/env bash

.PHONY: release-notes clean-local

# Default to the version file, but allow command-line overrides
VERSION ?= $(shell cat VERSION)

release-notes:
    @curr_ver=$(VERSION); \
    last_tag=$$(git tag -l '[0-9]*.[0-9]*.[0-9]*' --sort=-v:refname | head -n 1); \
    diffs="release-$$curr_ver.diffs"; \
    diff_list="release-$$curr_ver.lst"; \
    diff_tarball="release-$$curr_ver.tar.gz"; \
    echo "Comparing $$last_tag to current $$curr_ver..."; \
    git diff --no-ext-diff "$$last_tag" "$$curr_ver" > "$$diffs"; \
    git diff --name-only --diff-filter=AMR "$$last_tag" "$$curr_ver" > "$$diff_list"; \
    tar -cf - -T "$$diff_list" --transform "s|^|release-$$curr_ver/|" | gzip > "$$diff_tarball"; \
    ls -alrt release-$$curr_ver*

clean-local:
    @echo "Cleaning ephemeral release assets..."
    rm -f release-*.{tar.gz,lst,diffs}

Breaking Down the Recipe

• The Shell Choice (/usr/bin/env bash): We avoid hardcoding paths to ensure the script finds the correct Bash path on macOS, Linux, or inside a container.
• The Version Override (VERSION ?=): This allows the “pre-flight” trick: running make release-notes VERSION=HEAD to iterate on notes before you’ve actually tagged the release.
• Smart Tag Discovery (--sort=-v:refname): Using v:refname forces Git to use semantic versioning logic (so 1.10.0 correctly follows 1.2.0), while the glob pattern filters out “noisy” non-version tags.
• The Diff Filter (--diff-filter=AMR): This ensures the tarball only includes files that actually exist (Added, Modified, or Renamed). If a release deleted a file, this filter prevents tar from erroring out when it can’t find the missing file on disk.
• The Cleanup Crew (clean-local): remove the ephemeral artifacts using bash expansion

The AI Prompt

Once your assets are generated, upload them to AI and use a prompt that enforces your specific formatting standards.

The Prompt:

I have uploaded a file manifest (`.lst`), a unified diff
(`.diffs`), and a tarball of the source code (`.tar.gz`). Please
analyze these to identify the primary themes of this
release. Generate a release entry using this protocol:

1. Separate versions with 71 dashes.
2. Include a high-level executive summary paragraph.
3. Indent bullet points exactly 21 spaces to maintain a vertical column.
4. Group changes under category labels like `Core Bedrock:` or `Miniserver:`.
5. Provide plain-text ASCII only inside a code block.

The Result

In a recent dry run for Bedrock 3.8.8, this exact workflow allowed the AI to correctly identify that bedrock-miniserver.pl.in was refactored into Bedrock::MiniServer and Bedrock::MiniServer::CLI, while accurately noting that Bedrock::Error moved to the Bedrock-Core distribution.

In actuality, version 3.8.8 focuses on the primary goals of MiniServer maturation (more batteries included), Apache decoupling, and distribution hygiene. Here’s AI’s version:

-----------------------------------------------------------------------
Bedrock 3.8.8
-----------------------------------------------------------------------
This release focuses on the maturation of the MiniServer component, 
decoupling core Bedrock logic from Apache-specific dependencies, and 
improving distribution hygiene. Key architectural changes include 
refactoring the MiniServer into modular components and moving 
exception handling into the Bedrock-Core distribution.

2026-03-17 - 3.8.8 - MiniServer Maturation and Apache Decoupling
                     Miniserver:
                     - Refactored bedrock-miniserver.pl into modular 
                       Bedrock::MiniServer and Bedrock::MiniServer::CLI.
                     - Implemented zero-config scaffolding to 
                       automatically create application trees.
                     - Integrated full Bedrock configuration pipeline 
                       for parity with Apache environments.
                     - Updated bedrock_server_config to support both 
                       getter and setter operations.

                     Core:
                     - Moved Bedrock::Error and Bedrock::Exception to 
                       the Bedrock-Core distribution.
                     - Introduced Bedrock::FauxHandler as a production-
                       ready alias for test handlers.
                     - Added dist_dir() to BLM::Startup::Bedrock to 
                       expose distribution paths to templates.

                     Fixes:
                     - Demoted Apache-specific modules (mod_perl2, 
                       Apache2::Request) to optional recommendations.
                     - Improved Bedrock::Test::FauxHandler to handle 
                       caller-supplied loggers and safe destruction.

Conclusion

As I mentioned in a response to a recent Medium article, AI can be an accelerator for seasoned professionals. You’re not cheating. You did the work. AI does the wordsmithing. You edit, add color, and ship. What used to take 30 minutes now takes 3. Now that’s working smarter, not harder!

Pro-Tip

Add this to the top of your Makefile

SHELL := /usr/bin/env bash

# Default to the version file, but allow command-line overrides
VERSION ?= $(shell cat VERSION)

Copy this to a file named release-notes.mk

.PHONY: release-notes clean-local

release-notes:
    @curr_ver=$(VERSION); \
    last_tag=$$(git tag -l '[0-9]*.[0-9]*.[0-9]*' --sort=-v:refname | head -n 1); \
    diffs="release-$$curr_ver.diffs"; \
    diff_list="release-$$curr_ver.lst"; \
    diff_tarball="release-$$curr_ver.tar.gz"; \
    echo "Comparing $$last_tag to current $$curr_ver..."; \
    git diff --no-ext-diff "$$last_tag" "$$curr_ver" > "$$diffs"; \
    git diff --name-only --diff-filter=AMR "$$last_tag" "$$curr_ver" > "$$diff_list"; \
    tar -cf - -T "$$diff_list" --transform "s|^|release-$$curr_ver/|" | gzip > "$$diff_tarball"; \
    ls -alrt release-$$curr_ver*

clean-local:
    @echo "Cleaning ephemeral release assets..."
    rm -f release-*.{tar.gz,lst,diffs}

Then add release-notes.mk to your Makefile

include release-notes.mk

nginx is dead. Not metaphorically dead. Not “falling out of favor” dead. Actually, officially, put-a-date-on-it dead.

In November 2025 the Kubernetes project announced the retirement of Ingress NGINX — the controller running ingress for a significant fraction of the world’s Kubernetes clusters. Best-effort maintenance until March 2026. After that: no releases, no bugfixes, no security patches. GitHub repositories go read-only. Tombstone in place.

And before the body was even cold, we learned why. IngressNightmare — five CVEs disclosed in March 2025, headlined by CVE-2025-1974, rated 9.8 critical. Unauthenticated remote code execution. Complete cluster takeover. No credentials required. Wiz Research found over 6,500 clusters with the vulnerable admission controller publicly exposed to the internet, including Fortune 500 companies. 43% of cloud environments vulnerable. The root cause wasn’t a bug that could be patched cleanly - it was an architectural flaw baked into the design from the beginning. And the project that ran ingress for millions of production clusters was, in the end, sustained by one or two people working in their spare time.

Meanwhile Apache has been quietly running the internet for 30 years, governed by a foundation, maintained by a community, and looking increasingly like the adult in the room.

Let’s talk about how we got here.

Apache Was THE Web Server

Before we talk about what went wrong, let’s remember what Apache actually was. Not a web server. THE web server. At its peak Apache served over 70% of all websites on the internet. It didn’t win that position by accident - it won it by solving every problem the early web threw at it. Virtual hosting. SSL. Authentication. Dynamic content via CGI and then mod_perl. Rewrite rules. Per-directory configuration. Access control. Compression. Caching. Proxying. One by one, as the web evolved, Apache evolved with it, and the industry built on top of it.

Apache wasn’t just infrastructure. It was the platform on which the commercial internet was built. Every hosting provider ran it. Every enterprise deployed it. Every web developer learned it. It was as foundational as TCP/IP - so foundational that most people stopped thinking about it, the way you stop thinking about running water.

Then nginx showed up with a compelling story at exactly the right moment.

The Narrative That Stuck

The early 2000s brought a new class of problem - massively concurrent web applications, long-polling, tens of thousands of simultaneous connections. The C10K problem was real and Apache’s prefork MPM - one process per connection - genuinely struggled under that specific load profile. nginx’s event-driven architecture handled it elegantly. The benchmarks were dramatic. The config was clean and minimal, a breath of fresh air compared to Apache’s accumulated complexity. nginx felt modern. Apache felt like your dad’s car.

The “Apache is legacy” narrative took hold and never let go - even after the evidence for it evaporated.

Apache gained mpm_event, bringing the same non-blocking I/O and async connection handling that nginx was celebrated for. The performance gap on concurrent connections essentially closed. Then CDNs solved the static file problem at the architectural level - your static files live in S3 now, served from a Cloudflare edge node milliseconds from your user, and your web server never sees them. The two pillars of the nginx argument - concurrency and static file performance - were addressed, one by Apache’s own evolution and one by infrastructure that any serious deployment should be using regardless of web server choice.

But nobody reruns the benchmarks. The “legacy” label outlived the evidence by a decade. A generation of engineers learned nginx first, taught it to the next generation, and the assumption calcified into received wisdom. Blog posts from 2012 are still being cited as architectural guidance in 2025.

What Apache Does That nginx Can’t

Strip away the benchmark mythology and look at what these servers actually do when you need them to do something hard.

Apache’s input filter chain lets you intercept the raw request byte stream mid-flight - before the body is fully received - and do something meaningful with it. I’m currently building a multi-server file upload handler with real-time Redis progress tracking, proper session authentication, and CSRF protection implemented directly in the filter chain. Zero JavaScript upload libraries. Zero npm dependencies. Zero supply chain attack surface. The client sends bytes. Apache intercepts them. Redis tracks them. Done. nginx needs a paid commercial module to get close. Or you write C. Or you route around it to application code and wonder why you needed nginx in the first place.

Apache’s phase handlers let you hook into the exact right moment of the request lifecycle - post-read, header parsing, access control, authentication, response - each phase a precise intervention point. mod_perl embeds a full Perl runtime in the server with persistent state, shared memory, and pre-forked workers inheriting connection pools and compiled code across requests. mod_security gives you WAF capabilities your “modern” stack is paying a vendor for. mod_cache is a complete RFC-compliant caching layer that nginx reserves for paying customers.

And LDAP - one of the oldest enterprise authentication requirements there is. With mod_authnz_ldap it’s a few lines of config:

AuthType Basic
AuthName "Corporate Login"
AuthBasicProvider ldap
AuthLDAPURL ldap://ldap.company.com/dc=company,dc=com
AuthLDAPBindDN "cn=apache,dc=company,dc=com"
AuthLDAPBindPassword secret
Require ldap-group cn=developers,ou=groups,dc=company,dc=com

Connection pooling, SSL/TLS to the directory, group membership checks, credential caching - all native, all in config, no code required. With nginx you’re reaching for a community module with an inconsistent maintenance history, writing Lua, or standing up a separate auth service and proxying to it with auth_request - which is just mod_authnz_ldap reimplemented badly across two processes with an HTTP round trip in the middle.

Apache Includes Everything You’re Now Paying For

Look at Apache’s feature set and you’re reading the history of web infrastructure, one solved problem at a time. SSL termination? Apache had it before cloud load balancers existed to take it off your plate. Caching? mod_cache predates Redis by years. Load balancing? mod_proxy_balancer was doing weighted round-robin and health checks before ELB was a product. Compression, rate limiting, IP-based access control, bot detection via mod_security - Apache had answers to all of it before the industry decided each problem deserved its own dedicated service, its own operations overhead, and its own vendor relationship.

Apache didn’t accumulate features because it was undisciplined. It accumulated features because the web kept throwing problems at it and it kept solving them. The fact that your load balancer now handles SSL termination doesn’t mean Apache was wrong to support it - it means Apache was right early enough that the rest of the industry eventually built dedicated infrastructure around the same idea.

Now look at your AWS bill. CloudFront for CDN. ALB for load balancing and SSL termination. WAF for request filtering. ElastiCache for caching. Cognito for authentication. API Gateway for routing. Each one a line item. Each one a managed service wrapping functionality that Apache has shipped for free since before most of your team was writing code.

Amazon Web Services is, in a very real sense, Apache’s feature set repackaged as paid managed infrastructure. They looked at what the web needed, looked at what Apache had already solved, and built a business around operating those solutions at scale so you didn’t have to. That’s a legitimate value proposition - operations is hard and sometimes paying AWS is absolutely the right answer. But if you’re running a handful of servers and paying for half a dozen AWS services to handle concerns that Apache handles natively, maybe set the Wayback Machine to 2005, spin up Apache, and keep the credit card in your pocket.

Grandpa wasn’t just ahead of his time. Grandpa was so far ahead that Amazon built a cloud business catching up to him.

So Why Did You Choose nginx?

Be honest. The real reason is that you learned it first, or your last job used it, or a blog post from 2012 told you it was the modern choice. Maybe someone at a conference said Apache was legacy and you nodded along because everyone else was nodding. That’s how technology adoption works - narrative momentum, not engineering analysis.

But those nginx blinders have a cost. And the Kubernetes ecosystem just paid it in full.

The Cost of the nginx Blinders

The nginx Ingress Controller became the Kubernetes default early in the ecosystem’s adoption curve and the pattern stuck. Millions of production clusters. The de-facto standard. Fortune 500 companies. The Swiss Army knife of Kubernetes networking - and that flexibility was precisely its undoing.

The “snippets” feature that made it popular - letting users inject raw nginx config via annotations - turned out to be an unsanitizable attack surface baked into the design. CVE-2025-1974 exploited this to achieve unauthenticated RCE via the admission controller, giving attackers access to all secrets across all namespaces. Complete cluster takeover from anything on the pod network. In many common configurations the pod network is accessible to every workload in your cloud VPC. The blast radius was the entire cluster.

The architectural flaw couldn’t be fixed without gutting the feature that made the project worth using. So it was retired instead.

Here is the part nobody is saying out loud: Apache could have been your Kubernetes ingress controller all along.

The Apache Ingress Controller exists. It supports path and host-based routing, TLS termination, WebSocket proxying, header manipulation, rate limiting, mTLS - everything Ingress NGINX offered, built on a foundation with 30 years of security hardening and a governance model that doesn’t depend on one person’s spare time. It doesn’t have an unsanitizable annotation system because Apache’s configuration model was designed with proper boundaries from the beginning. The full Apache module ecosystem - mod_security, mod_authnz_ldap, the filter chain, all of it - available to every ingress request.

The Kubernetes community never seriously considered it. nginx had the mindshare, nginx got the default recommendation, nginx became the assumed answer before the question was even finished. Apache was dismissed as grandpa’s web server by engineers who had never actually used it for anything hard - and so the ecosystem bet its ingress layer on a project sustained by volunteers and crossed its fingers.

The nginx blinders cost the industry IngressNightmare, 6,500 exposed clusters, and a forced migration that will consume engineering hours across thousands of organizations in 2026. Not because Apache wasn’t available. Because nobody looked.

nginx is survived by its commercial fork nginx Plus, approximately 6,500 vulnerable Kubernetes clusters, and a generation of engineers who will spend Q1 2026 migrating to Gateway API - a migration they could have avoided entirely.

Who’s Keeping The Lights On

Here’s the conversation that should happen in every architecture review but almost never does: who maintains this and what happens when something goes wrong?

For Apache the answer has been the same for over 30 years. The Apache Software Foundation - vendor-neutral, foundation-governed, genuinely open source. Security vulnerabilities found, disclosed responsibly, patched. A stable API that doesn’t break your modules between versions. Predictable release cycles. Institutional stability that has outlasted every company that ever tried to compete with it.

nginx’s history is considerably more complicated. Written by Igor Sysoev while employed at Rambler, ownership murky for years, acquired by F5 in 2019. Now a critical piece of infrastructure owned by a networking hardware vendor whose primary business interests may or may not align with the open source project. nginx Plus - the version with the features that actually compete with Apache on a level playing field - is commercial. OpenResty, the variant most people reach for when they need real programmability, is a separate project with its own maintenance trajectory.

The Ingress NGINX project had millions of users and a maintainership you could count on one hand. That’s not a criticism of the maintainers - it’s an indictment of an ecosystem that adopted a critical infrastructure component without asking who was keeping the lights on.

Three decades of adversarial testing by the entire internet is a security posture no startup’s stack can match. The Apache Software Foundation will still be maintaining Apache httpd when the company that owns your current stack has pivoted twice and been acqui-hired into oblivion.

Long Live Apache

The engineers who dismissed Apache as legacy were looking at a 2003 benchmark and calling it a verdict. They missed the server that anticipated every problem modern infrastructure is still solving, that powered the internet before AWS existed to charge you for the privilege, and that was sitting right there in the Kubernetes ecosystem waiting to be evaluated while the community was busy betting critical infrastructure on a volunteer project with an architectural time bomb in its most popular feature.

Grandpa didn’t just know what he was doing. Grandpa was building the platform you’re still trying to reinvent - badly, in JavaScript, with a vulnerability disclosure coming next Tuesday and a maintainer burnout announcement the Tuesday after that.

The server is fine. It was always fine. Touch grass, update your mental model, and maybe read the Apache docs before your next architecture meeting.

RIP nginx Ingress Controller. 2015-2026. Maintained by one guy in his spare time. Missed by the 43% of cloud environments that probably should have asked more questions.

Sources

• IngressNightmare - CVE details and exposure statistics Wiz Research, March 24, 2025 https://www.wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities
• Ingress NGINX Retirement Announcement Kubernetes SIG Network and Security Response Committee, November 11, 2025 https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/
• Ingress NGINX CVE-2025-1974 - Official Kubernetes Advisory Kubernetes, March 24, 2025 https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/
• Transitioning Away from Ingress NGINX - Maintainership and architectural analysis Google Open Source Blog, February 2026 https://opensource.googleblog.com/2026/02/the-end-of-an-era-transitioning-away-from-ingress-nginx.html
• F5 Acquisition of nginx F5 Press Release, March 2019 https://www.f5.com/company/news/press-releases/f5-acquires-nginx-to-bridge-netops-and-devops

Disclaimer: This article was written with AI assistance during a long discussion on the features and history of Apache and nginx, drawing on my experience maintaining and using Apache over the last 20+ years. The opinions, technical observations, and arguments are entirely my own. I am in no way affiliated with the ASF, nor do I have any financial interest in promoting Apache. I have been using and benefiting from Apache since 1998 and continue to discover features and capabilities that surprise me even to this day.

This is the last in a 3 part series on Scriptlets. You can catch up by reading our introduction and dissection of Scriptlets.

• Part I - An introduction to scriptlets
• Part II - The anatomy of a scriptlet

In this final part, we talk about restraint - the discipline that keeps a clever trick from turning into a maintenance hazard.

That uneasy feeling…

So you are starting to write a few scriptlets and it seems pretty cool. But something doesn’t feel quite right…

You’re editing a Makefile and suddenly you feel anxious. Ah, you expected syntax highlighting, linting, proper indentation, and maybe that warm blanket of static analysis. So when we drop a 20 - line chunk of Perl or Python into our Makefile, our inner OCD alarms go off. No highlighting. No linting. Just raw text.

The discomfort isn’t a flaw - it’s feedback. It tells you when you’ve added too much salt to the soup.

A scriptlet is not a script!

A scriptlet is a small, focused snippet of code embedded inside a Makefile that performs one job quickly and deterministically. The “-let” suffix matters. It’s not a standalone program. It’s a helper function, a convenience, a single brushstroke that belongs in the same canvas as the build logic it supports.

If you ever feel the urge to bite your nails, pick at your skin, or start counting the spaces in your indentation - stop. You’ve crossed the line. What you’ve written is no longer a scriptlet; it’s a script. Give it a real file, a shebang, and a test harness. Keep the build clean.

Why we use them

Scriptlets shine where proximity and simplicity matter more than reuse (not that we can’t throw it in a separate file and include it in our Makefile).

• Cleanliness: prevents a recipe from looking like a shell script.
• Locality: live where they’re used. No path lookups, no installs.
• Determinism: transform well-defined input into output. Nothing more.
• Portability (of the idea): every CI/CD system that can run make can run a one-liner.

A Makefile that can generate its own dependency file, extract version numbers, or rewrite a cpanfile doesn’t need a constellation of helper scripts. It just needs a few lines of inline glue.

Why they’re sometimes painful

We lose the comforts that make us feel like professional developers:

• No syntax highlighting.
• No linting or type hints.
• No indentation guides.
• No “Format on Save.”

The trick is to accept that pain as a necessary check on the limits of the scriptlet. If you’re constantly wishing for linting and editor help, it’s your subconscious telling you: this doesn’t belong inline anymore. You’ve outgrown the -let.

When to promote your scriplet to a script…

Promote a scriptlet to a full-blown script when:

• It exceeds 30-50 lines.
• It gains conditionals or error handling.
• You need to test it independently.
• It uses more than 1 or 2 non-core features.
• It’s used by more than one target or project.
• You’re debugging quoting more than logic.
• You’re spending more time fixing indentation, than working on the build

At that point, you’re writing software, not glue. Give it a name, a shebang, and a home in your tools/ directory.

When to keep it inside your Makefile…

Keep it inline when:

• It’s short, pure, and single-use.
• It depends primarily on the environment already assumed by your build (Perl, Python, awk, etc.).
• It’s faster to read than to reference.

A good scriptlet reads like a make recipe: do this transformation right here, right now.

define create_cpanfile =
    while (<STDIN>) {
        s/[#].*//; s/^\s+|\s+$//g; next if $_ eq q{};
        my ($mod,$v) = split /\s+/, $_, 2;
        print qq{requires "$mod", "$v";\n};
    }
endef

export s_create_cpanfile = $(value create_cpanfile)

That’s a perfect scriptlet: small, readable, deterministic, and local.

Rule of Thumb: If it fits on one screen, keep it inline. If it scrolls, promote it.

Tools for the OCD developer

If you must relieve the OCD symptoms without promotion of your scriptlet to a script…

• Add a lint-scriptlets target: perl -c -e '$(s_create_requires)' checks syntax without running it.
• Some editors (Emacs mmm-mode, Vim polyglot) can treat marked sections as sub-languages to enable localized language specific editing features.
• Use include to include a scriptlet into your Makefile

…however try to resist the urge to over-optimize the tooling. Feeling the uneasiness grow helps identify the boundary between scriptlets and scripts.

You’ve been warned!

Because scriptlets are powerful, flexible, and fast, it’s easy to reach for them too often or make them the focus of your project. They start as a cure for friction - a way to express a small transformation inline - but left unchecked, they can sometimes grow arms and legs. Before long, your Makefile turns into a Frankenstein monster.

The great philosopher Basho (or at least I think it was him) once said:

A single aspirin tablet eases pain. A whole bottle sends you to the hospital.

Thanks for reading.

Learn More

• GNU make
• value
• define
• export

In our previous blog post “Go Ahead ‘make’ My Day” we presented the scriptlet, an advanced make technique for spicing up your Makefile recipes. In this follow-up, we’ll deconstruct the scriptlet and detail the ingredients that make up the secret sauce.

Introducing the Scriptlet

Makefile scriptlets are an advanced technique that uses GNU make’s powerful functions to safely embed a multi-line script (Perl, in our example) into a single, clean shell command. It turns a complex block of logic into an easily executable template.

An Example Scriptlet

#-*- mode: makefile; -*-

DARKPAN_TEMPLATE="https://cpan.openbedrock.net/orepan2/authors/D/DU/DUMMY/%s-%s.tar.gz"

define create_requires =
 # scriptlet to create cpanfile from an list of required Perl modules
 # skip comments
 my $DARKPAN_TEMPLATE=$ENV{DARKPAN_TEMPLATE};

 while (s/^#[^\n]+\n//g){};

 # skip blank lines
 while (s/\n\n/\n/) {};

 for (split/\n/) { 
  my ($mod, $v) = split /\s+/;
  next if !$mod;

  my $dist = $mod;
  $dist =~s/::/\-/g;

  my $url = sprintf $DARKPAN_TEMPLATE, $dist, $v;

  print <<"EOF";
requires \"$mod\", \"$v\",
  url => \"$url\";
EOF
 }

endef

export s_create_requires = $(value create_requires)

cpanfile.darkpan: requires.darkpan
    DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE); \
    DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl -0ne "$$s_create_requires" $< > $@ || rm $@

Dissecting the Scriptlet

1. The Container: Defining the Script (define / endef)

This section creates the multi-line variable that holds your entire Perl program.

define create_requires =
# Perl code here...
endef

• define ... endef: This is GNU Make’s mechanism for defining a recursively expanded variable that spans multiple lines. The content is not processed by the shell yet; it’s simply stored by make.
• The Advantage: This is the only clean way to write readable, indented code (like your while loop and if statements) directly inside a Makefile.

2. The Bridge: Passing Environment Data (my $ENV{...})

This is a critical step for making your script template portable and configurable.

my $DARKPAN_TEMPLATE=$ENV{DARKPAN_TEMPLATE};

• The Problem: Your Perl script needs dynamic values (like the template URL) that are set by make.
• The Solution: Instead of hardcoding the URL, the Perl code is designed to read from the shell environment variable $ENV{DARKPAN_TEMPLATE}. This makes the script agnostic to its calling environment, delegating the data management back to the Makefile.

3. The Transformer: Shell Preparation (export and $(value))

This is the “magic” that turns the multi-line Make variable into a single, clean shell command.

export s_create_requires = $(value create_requires)

• $(value create_requires): This is a specific Make function that performs a direct, single-pass expansion of the variable’s raw content. Crucially, it converts the entire multi-line block into a single-line string suitable for export, preserving special characters and line breaks that the shell will execute.
• export s_create_requires = ...: This exports the multi-line Perl script content as an environment variable (s_create_requires) that will be accessible to any shell process running in the recipe’s environment.

4. The Execution: Atomic Execution ($$ and perl -0ne)

The final recipe executes the entire, complex process as a single, atomic operation, which is the goal of robust Makefiles.

cpanfile.darkpan: requires.darkpan
    DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE); \
    DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl -0ne "$$s_create_requires" $< > $@ || rm $@

• DARKPAN_TEMPLATE=$(DARKPAN_TEMPLATE): This creates the local shell variable.
• DARKPAN_TEMPLATE=$$DARKPAN_TEMPLATE perl...: This is the clean execution. The first DARKPAN_TEMPLATE= passes the newly created shell variable’s value as an environment variable to the perl process. The $$ ensures the shell variable is properly expanded before the Perl interpreter runs it.
• perl -0ne "...": Runs the Perl script:

  * `-n` and `-e` (Execute script on input)
  * `-0`: Tells Perl to read the input as one single block
    (slurping the file), which is necessary for your multi-line
    regex and `split/\n/` logic.
  

• || rm $@: This is the final mark of quality. It makes the entire command transactional—if the Perl script fails, the half-written target file ($@) is deleted, forcing make to try again later.

Hey Now! You’re a Rockstar!

(..get your game on!)

Mastering build automation using make will transform you from being an average DevOps engineer into a rockstar. GNU make is a Swiss Army knife with more tools than you might think! The knives are sharp and the tools are highly targeted to handle all the real-world issues build automation has encountered over the decades. Learning to use make effectively will put you head and shoulders above the herd (see what I did there? 😉).

Calling All Pythonistas!

The scriptlet technique creates a powerful, universal pattern for clean, atomic builds:

• It’s Language Agnostic: Pythonistas! Join the fun! The same define/export technique works perfectly with python -c.
• The Win: This ensures that every developer - regardless of their preferred language - can achieve the same clean, atomic build and avoid external script chaos.

Learn more about GNU make and move your Makefiles from simple shell commands to precision instruments of automation.

Thanks for reading.

Learn More

• GNU make
• value
• define
• export

While looking at some old bash script that bumps my semantic versions I almost puked looking at my old ham handed way of bumping the version. That led me to see how I could do it “better”. Why? I dunno know…bored on a Saturday morning and not motivated enough to do the NY Times crossword…

So you want to bump a semantic version string like 1.2.3 - major, minor, or patch - and you don’t want ceremony. You want one line, no dependencies, and enough arcane flair to scare off coworkers.

Here’s a single-line Bash–Perl spell that does exactly that:

v=$(cat VERSION | p=$1 perl -a -F[.] -pe \
'$i=$ENV{p};$F[$i]++;$j=$i+1;$F[$_]=0 for $j..2;$"=".";$_="@F"')

What It Does

• Reads the current version from a VERSION file (1.2.3)
• Increments the part you pass (0 for major, 1 for minor, 2 for patch)
• Resets all lower parts to zero
• Writes the result to v

Scriptlet Form

Wrap it like this in a shell function:

bump() {
  v=$(cat VERSION | p=$1 perl -a -F[.] -pe \
  '$i=$ENV{p};$F[$i]++;$j=$i+1;$F[$_]=0 for $j..2;$"=".";$_="@F"')
  echo "$v" > VERSION
}

Then run:

bump 2   # bump patch (1.2.3 => 1.2.4)
bump 1   # bump minor (1.2.3 => 1.3.0)
bump 0   # bump major (1.2.3 => 2.0.0)

Makefile Integration

Want to bump right from make?

bump-major:
    @v=$$(cat VERSION | p=0 perl -a -F[.] -pe '$$i=$$ENV{p};$$F[$$i]++;$$j=$$i+1;$$F[$$_]=0 for $$j..2;$$"=".";$_="$$F"') && \
    echo $$v > VERSION && echo "New version: $$v"

bump-minor:
    @$(MAKE) bump-major p=1

bump-patch:
    @$(MAKE) bump-major p=2

Or break it out into a .bump-version script and source it from your build tooling.

How It Works (or …Why I Love Perl)

-a            # autosplit into @F
-F[.]         # split on literal dot
$i=$ENV{p}    # get part index from environment (e.g., 1 for minor)
$F[$i]++      # bump it
$j=$i+1       # start index for resetting
$F[$_]=0 ...  # zero the rest
$"=".";       # join array with dots
$_="@F"       # set output

If you have to explain this to some junior dev, just say RTFM skippy perldoc perlrun. Use the force Luke.

And if the senior dev wags his finger and say UUOC, tell him Ego malum edo.

test test check check

An "END" code block is executed as late as possible, that is, after perl
has finished running the program and just before the interpreter is
being exited, even if it is exiting as a result of a die() function.
(But not if it's morphing into another program via "exec", or being
blown out of the water by a signal--you have to trap that yourself (if
you can).) You may have multiple "END" blocks within a file--they will
execute in reverse order of definition; that is: last in, first out
(LIFO). "END" blocks are not executed when you run perl with the "-c"
switch, or if compilation fails....

Perl’s END blocks are useful inside your script for doing things like cleaning up after itself, closing files or disconnecting from databases. In many cases you use an END block to guarantee certain behaviors like a commit or rollback of a transaction. You’ll typically see END blocks in scripts, but occassionally you might find one in a Perl module.

Over the last four years I’ve done a lot of maintenance work on legacy Perl applications. I’ve learned more about Perl in these four years than I learned in the previous 20. Digging into bad code is the best way to learn how to write good code. It’s sometimes hard to decide if code is good or bad but to paraphrase a supreme court justice, I can’t always define bad code, but I know it when I see it.

One of the gems I’ve stumbled upon was a module that provided needed functionality for many scripts that included its own END block.

Putting an END block in a Perl module is an anti-pattern and just bad mojo. A module should never contain an END block. Here are some alternatives:

• If cleanup is necessary, provide (and document) a cleanup() method
• Use a destructor (DESTROY) in an object-oriented module

Modules should provide functionality - not take control of your script’s shutdown behavior.

Why Would Someone Put an END Block in a Perl Module?

The first and most obvious answer is that they were unaware of how DESTROY blocks can be employed. If you know something about the author and you’re convinced they know better, then why else?

I theorize that the author was trying to create a module that would encapsulate functionality that he would use in EVERY script he wrote for the application. While the faux pas might be forgiven I’m not ready to put my wagging finger back in its holster.

If you want to write a wrapper for all your scripts and you’ve already settled on using a Perl module to do so, then please for all that is good and holy do it right. Here are some potential guidelines for that wrapper:

• A new() method that instantiates your wrapper
• An init() method that encapsulates the common startup operations with options to control whether some are executed or not
• A run() method that executes the functionality for the script
• A finalize() method for executing cleanup procedures
• POD that describes the common functionality provided as well as any options for controlling their invocation

All of these methods could be overridden if you just use a plain ‘ol Perl module. For those that prefer composition over inheritance, use a role with something like Role::Tiny to provide those universally required methods. Using Role::Tiny provides better flexibility by allowing you to use those methods before or after your modifications to their behavior.

How Can We Take Back Control?

The particular script I was working on included a module(whose name I shall not speak) that included such an END block. My script should have exited cleanly and quietly. Instead, it produced mysterious messages during shutdown. Worse yet I feared some undocumented behaviors and black magic might have been conjured up during that process! After a bit of debugging, I found the culprit:

• The module had an END block baked into it
• This END block printed debug messages to STDERR while doing other cleanup operations
• Worse, it ran unconditionally when my script terminated

The Naive Approach

My initial attempt to suppress the module’s END block:

END {
    use POSIX;
    POSIX::_exit(0);
}

This works as long as my script exits normally. But if my script dies due to an error, the rogue END block still executes. Not exactly the behavior I want.

A Better Approach

Here’s what I want to happen:

• Prevent any rogue END block from executing.
• Handle errors gracefully.

• Ensure my script always exits with a meaningful status code.

• A better method for scripts that need an END block to claw back control:

use English qw(-no_match_vars);
use POSIX ();

my $retval = eval {
    return main();
};

END {
  if ($EVAL_ERROR) {
      warn "$EVAL_ERROR";  # Preserve exact error message
  }

  POSIX::_exit( $retval // 1 );
}

• Bypasses all END blocks – Since POSIX::_exit() terminates the process immediately
• Handles errors cleanly – If main() throws an exception, we log it without modifying the message
• Forces explicit return values – If main() forgets to return a status, we default to 1, ensuring no silent failures.
• Future maintainers will see exactly what’s happening

Caveat Emptor

Of course, you should know what behavior you are bypassing if you decide to wrestle control back from some misbehaving module. In my case, I knew that the behaviors being executed in the END block could safely be ignored. Even if they couldn’t be ignored, I can still provide those behaviors in my own cleanup procedures.

Isn’t this what future me or the poor wretch tasked with a dumpster dive into a legacy application would want? Explicitly seeing the whole shebang without hours of scratching your head looking for mysterious messages that emanate from the depths is priceless. It’s gold, Jerry! Gold!

Next up…a better wrapper.

In our previous posts, we explored how Apache 2.4 changed its handling of directory requests when DirectorySlash Off is set, breaking the implicit /dir → /dir/ redirect behavior that worked in Apache 2.2. We concluded that while an external redirect is the only reliable fix, this change in behavior led us to an even bigger question:

Is this a bug or an intentional design change in Apache?

After digging deeper, we’ve uncovered something critically important that is not well-documented:

Apache does not restart the request cycle after an internal rewrite, and this can break expected behaviors like DirectoryIndex.

This post explores why this happens, whether it’s a feature or a bug, and why Apache’s documentation should explicitly clarify this behavior.

What Happens When Apache Internally Rewrites a Request?

Let’s revisit the problem: we tried using an internal rewrite to append a trailing slash for directory requests:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ /$1/ [L]

Expected Behavior: - Apache should internally rewrite /setup to /setup/. - Since DirectoryIndex index.roc is set, Apache should serve index.roc.

Actual Behavior: - Apache internally rewrites /setup to /setup/, but then immediately fails with a 403 Forbidden. - The error log states: AH01276: Cannot serve directory /var/www/vhosts/treasurersbriefcase/htdocs/setup/: No matching DirectoryIndex (none) found - Apache is treating /setup/ as an empty directory instead of recognizing index.roc.

The Key Issue: Apache Does Not Restart Request Processing After an Internal Rewrite

Unlike what many admins assume, Apache does not start over after an internal rewrite.

How Apache Processes Requests (Simplified)

1. The request arrives (/setup).
2. Apache processes mod_rewrite.
3. Apache determines how to serve the request.
4. If an index file (index.html, index.php, etc.) exists, DirectoryIndex resolves it.

Why Internal Rewrites Don’t Restart Processing

• Apache processes mod_rewrite before it checks DirectoryIndex.
• Once a rewrite occurs, Apache continues processing from where it left off.
• This means it does not re-check DirectoryIndex after an internal rewrite.
• Instead, it sees /setup/ as an empty directory with no default file and denies access with 403 Forbidden.

Is This a Bug or a Feature?

First, let’s discuss why this worked in Apache 2.2.

The key reason internal rewrites worked in Apache 2.2 is that Apache restarted the request processing cycle after a rewrite. This meant that:

• After an internal rewrite, Apache treated the rewritten request as a brand-new request.
• As a result, it re-evaluated DirectoryIndex and correctly served index.html, index.php, or any configured default file.
• Since DirectorySlash was handled earlier in the request cycle, Apache 2.2 still applied directory handling rules properly, even after an internal rewrite.

In Apache 2.4, this behavior changed. Instead of restarting the request cycle, Apache continues processing the request from where it left off. This means that after an internal rewrite, DirectoryIndex is never reprocessed, leading to the 403 Forbidden errors we encountered. This fundamental change explains why no internal solution works the way it did in Apache 2.2.

Why It’s Likely an Intentional Feature

• In Apache 2.2, some rewrites did restart the request cycle, which was seen as inefficient.
• In Apache 2.4, request processing was optimized for performance, meaning it does not restart after an internal rewrite.
• The behavior is consistent across different Apache 2.4 installations.
• Some discussions in Apache’s mailing lists and bug tracker mention this as “expected behavior.”

Why This is Still a Problem

• This behavior is not explicitly documented in mod_rewrite or DirectoryIndex docs.
• Most admins expect Apache to reprocess the request fully after a rewrite.
• The lack of clarity leads to confusion and wasted debugging time.

Implications for Apache 2.4 Users

1. Mod_Rewrite Behavior is Different From What Many Assume

• Internal rewrites do not restart request processing.
• DirectoryIndex is only evaluated once, before the rewrite happens.
• This is not obvious from Apache’s documentation.

2. The Only Reliable Fix is an External Redirect

Since Apache won’t reprocess DirectoryIndex, the only way to guarantee correct behavior is to force a new request via an external redirect:

RewriteEngine On
RewriteCond %{REQUEST_URI} !/$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} -d
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1/ [R=301,L]

This forces Apache to start a completely new request cycle, ensuring that DirectoryIndex is evaluated properly.

3. Apache Should Improve Its Documentation

We believe this behavior should be explicitly documented in: - mod_rewrite documentation (stating that rewrites do not restart request processing). - DirectoryIndex documentation (noting that it will not be re-evaluated after an internal rewrite).

This would prevent confusion and help developers troubleshoot these issues more efficiently.

Conclusion: A Feature, But a Poorly Documented One

• The fact that Apache does not restart processing after an internal rewrite is likely an intentional design choice.
• However, this is not well-documented, leading to confusion.
• The only solution remains an external redirect to force a fresh request cycle.
• We believe Apache should update its documentation to reflect this behavior more clearly.