Credit: The caravan of Perl AWS support

Today I’m releasing version 2.8.0 of Amazon::API.

This version of Amazon::API is a significant milestone. It represents the first release in a series of planned releases that seeks to achieve parity with the Python Botocore library.

Botocore library parity means that Amazon::API can consume the same Botocore service metadata and implement the same low-level AWS client behaviors needed to construct, sign, send, and interpret requests correctly.

It does not mean reproducing every convenience API exposed by Boto3. Features such as waiters, transfer managers, resources, and higher-level helpers only require that Amazon::API expose the underlying primitives needed to implement them.

Over the last year, with the assistance of AI, I’ve been able to accelerate development of my lightweight interface to Amazon’s APIs. The existing model-driven architecture of Amazon::API, already built around Botocore metadata, is now approaching the AWS SDK for Perl I imagined in 2017.

The result is a Perl AWS API that is much closer to the capabilities of the Python Botocore library. It also moves Perl closer to the broader goal of first-class AWS support that the community has been pursuing since Paws was released in 2015.

How AI Is Accelerating the Work

Botocore parity is a daunting task for a Perl developer.

The challenge comes from the sheer number of Botocore behaviors that one developer must recognize and implement. Those implementations then have to be tested. The places where they differ from Boto3 results must be analyzed by tracing through metadata and code. Each discovery must finally be turned into a test and a fix. AI tools such as Claude and ChatGPT have made it practical for me to work through that surface area much more quickly, while I continue to make the architectural and design decisions that determine what Amazon::API should become.

AI has been especially useful in the less glamorous but time-consuming parts of the work: debugging serialization problems, improving documentation, and writing or expanding unit tests. Those are exactly the areas where a second set of eyes, rapid iteration, and the ability to compare behavior across a large codebase can accelerate development.

It has also slowed things down at times. AI can hallucinate interfaces that do not exist, spend too much time nit-picking perfectly reasonable constructions or proposing paranoid guardrails, and head down dead ends when important context has been lost or forgotten. Using it effectively still requires knowing the codebase and recognizing when a suggestion is technically incorrect or runs counter to the architectural philosophy underpinning the entire work. A developer must be willing to question it and push back hard before AI sends one down a rabbit hole.

AI has also been extremely useful in helping me understand the broader AWS ecosystem. Implementing an API operation requires knowing how to construct the request correctly, but that is very different from understanding why a service behaves the way it does or what problem a particular capability is intended to solve. Endpoint resolution is a good example: implementing the rules is one task; understanding the use cases behind regional, account-specific, ARN-derived, or service-specific endpoints is another. AI has helped bridge that gap between implementing AWS behavior and understanding the ecosystem that gave rise to it.

AI’s limitations are real, but so are the gains. On balance, AI has accelerated this work far more than it has hindered it.

Why Another AWS API for Perl? Paws Already Exists.

Amazon::API was not created in response to Paws. Paws had already been on CPAN for a couple of years when I began formal development of Amazon::API in 2017, but Paws was still very much a project under active development. Its own documentation cautioned that the SDK should not yet be considered stable, even though it was already being used in production.

I was solving a similar problem independently, with a different set of design goals. Amazon::API itself did not reach CPAN until December of 2018.

Amazon::API was not an attempt to reinvent the wheel. It’s a different wheel that was taking shape while Paws was still evolving.

A Different Approach

Although Paws and Amazon::API attempt to solve the same problem, they take different approaches. Amazon::API started light and remains that way today. It keeps the runtime small, keeps AWS service definitions independently updatable, and returns ordinary Perl data instead of building a large generated object model around every service.

That separation is particularly important in an ecosystem where AWS service definitions change constantly. A new operation, shape, or pagination rule does not necessarily require a new Amazon::API release. Individual service distributions can instead be regenerated against newer Botocore metadata, allowing the AWS API definitions to remain current without coupling their release cycle to that of the runtime.

A Common Foundation

Paws and Amazon::API both make use of Botocore metadata, but they diverge sharply in how they package and expose it. Paws generates a broad Perl object model representing AWS services, requests, responses, and associated types. Amazon::API uses generated service definitions with a comparatively small runtime and normally returns plain Perl data structures.

Because those generated service distributions are independent of the runtime, an application can install only the AWS services it actually needs and update those definitions independently as AWS evolves.

Amazon::API is therefore aimed at Perl applications where a lightweight, data-oriented interface, independently current AWS service definitions, and a small production footprint are important.

Amazon::API 2.8.0 - Current Status

Version 2.8.0 increases the stability and robustness of the API. It includes serialization fixes, a suite of tests that exercise the full set of Botocore-derived protocol and pagination test suites, endpoint rule-set resolution, and updated documentation. Additionally, features and capability gaps that still exist have been articulated in a formal roadmap.

The result is a much clearer picture of what Amazon::API supports today, what remains to reach broader Botocore parity, and what is intentionally outside the scope of the core runtime. The table below summarizes that current state.

Capability Status Notes
Service/operation model support Implemented Generated from Botocore service metadata; independently updatable
Request serialization/deserialization Implemented Existing protocols supported; Smithy RPC protocols are planned separately
Endpoint rule-set resolution Implemented 2.8.0 milestone; modeled endpoint context and auth metadata supported for SigV4
SigV4 signing Implemented Includes modeled signing name, signing region, and disableDoubleEncoding
SigV4a Planned (2.9.0) Required for broader modern AWS endpoint/auth parity
S3 compatibility Planned Much closer after 2.8.0; full support depends on remaining auth and service-specific behavior
S3 Express Planned (2.10.0) Requires session auth and v4-s3express support
Smithy RPC protocols Planned (3.0.0) Includes rpcv2Cbor and related protocol behavior
Pagination Implemented Model-driven/compiler-backed
Checksums / payload-signing behavior Planned (3.0.0) Requires parity audit against Botocore handlers and protocol traits
Streaming / event streams Planned (3.0.0) Needed for applicable Smithy and streaming operations
Botocore handler semantics Planned (3.0.0) Systematic audit needed to identify remaining request/response mutations
Credential provider behavior Out of scope Owned by Amazon::Credentials; parity should be evaluated separately
Retry policy Out of scope Execution policy rather than request/protocol correctness
Waiters / transfer helpers / resources Out of scope Amazon::API should expose the primitives needed to implement them above the core runtime

Tentative Roadmap

The remaining work is being broken into a small number of focused releases, each intended to close a specific set of gaps on the way toward broader Botocore parity.

Version Feature/Capability Target Date Status
2.8.0 Endpoint rule sets + modeled SigV4 signing 9/26 Released
2.9.0 SigV4a 10/1 -
2.10.0 S3 Express 10/31 -
3.0.0 Smithy protocol work + handler audit/fixes 12/31 -

Testing Status

Amazon::API is tested at several different levels because no single test strategy is sufficient for a runtime that interprets Botocore service metadata.

The ordinary unit and regression suite exercises specific runtime behavior: request construction, serialization, deserialization, pagination, URI handling, endpoint context evaluation, service hooks, and signing.

A second layer uses protocol test cases derived from the Botocore test corpus. These tests provide known request and response examples across the AWS protocol families and allow Amazon::API behavior to be checked against expectations derived from Botocore.

The corpus is not treated as exhaustive. When live AWS testing exposed a behavior or edge case that was not represented by the existing corpus, I added a focused regression test or custom corpus case.

Finally, selected requests are validated against live AWS services. This is particularly useful for endpoint resolution and signing, where a request can appear correct locally but still be rejected by AWS.

The current 2.8.0 testing picture looks like this:

Test Area Status Notes
Unit and regression suite Passing Covers core runtime behavior including serialization, deserialization, pagination, request construction, URI handling, endpoint context, service hooks, and signing
Botocore protocol corpus Passing 671 tests across 15 protocol test files
Pagination Passing Dedicated coverage for model-driven paginator compilation and runtime evaluation
Endpoint rule-set evaluation Passing Covers rule evaluation, endpoint context sources and precedence, scalar template handling, and service endpoint selection
S3Control endpoint and URI behavior Passing Covers ARN-driven endpoint context, region selection, and percent encoding of non-greedy URI labels
S3 request construction Passing Covers virtual-hosted and force-path-style addressing, the S3 service hook, and final request URL composition
S3 greedy URI handling Passing Covers {Key+} behavior, preserving / while percent encoding other reserved characters
S3 response deserialization Passing Covers the rest-xml root-unwrapping behavior that previously affected some S3 responses
Kinesis endpoint resolution Passing ListShards with StreamARN resolves to the account-specific control endpoint
SigV4 auth propagation Passing Endpoint-derived signing name, signing region, and disableDoubleEncoding are propagated through the signing stack
Live AWS validation Selective Used for endpoint and signing behavior where AWS itself provides the final validation
Custom regression coverage Ongoing Added whenever real service behavior exposes a case not represented by the existing Botocore corpus

Passing tests do not prove that every AWS operation is correct. What they do provide is a repeatable way to increase confidence and coverage: Botocore-derived corpus tests establish broad protocol behavior, focused regressions preserve specific fixes and service behaviors, and live AWS calls validate the cases where the service itself is the final authority.

Why a Current Perl API for AWS Matters

There is still a larger question: why am I investing this much effort in a Perl API for AWS when many Perl developers have moved on to other languages with more complete and current support for AWS?

First, let’s discuss the elephant in the room. Isn’t Perl dying a slow death? I don’t think so. In fact, I think just the opposite might happen, especially if the community continues to enhance its toolchains to keep up with security and advancing technology. I simply refuse to accept the premise that Perl is simply fading away.

AI may be another catalyst that creates something of a renaissance for Perl, especially as it helps refresh, secure, and enhance legacy Perl-based applications. Large bodies of working Perl code that were once expensive or risky to modernize can now be analyzed, refactored, tested, documented, and secured with much greater leverage than was possible even a few years ago. A legacy Perl application still running today is often doing something important enough that replacing it is neither simple nor inexpensive.

And then there are the selfish reasons to build a modern AWS API for Perl. I need a capable, complete Perl AWS SDK. The applications and infrastructure I maintain need more than I can comfortably get from the existing choices, and rewriting those systems in another language simply to gain better AWS support is not an attractive solution.

And finally, I’d like to share my work. Perl has always benefited from developers publishing the tools they built to solve their own problems. Much of the software I rely on exists because someone else chose to do exactly that. Amazon::API is my contribution back to that tradition.

Why You Should Give Amazon::API a Try

If you’ve used Paws or other Perl interfaces to AWS, you may still be wondering what makes Amazon::API worth trying.

Your answer is probably specific to your needs. If you need:

  • a small runtime that works well in containerized applications
  • independently generated AWS service definitions, so your application installs only what it needs
  • ordinary Perl data structures for requests and responses without the overhead of a heavy framework
  • developer-friendly documentation that is separate from the production runtime
  • service definitions that can keep up with Botocore changes
  • an actively maintained Perl-based AWS SDK

…then Amazon::API may just fit the bill.

What’s Next

Version 2.8.0 is an important milestone, but it is not the end of the work.

The immediate roadmap is to continue closing the remaining gaps between Amazon::API and Botocore:

  • SigV4a
  • S3 Express support
  • Smithy protocol support

…and, of course, to continue auditing Botocore behavior, expanding test coverage, and fixing defects as they are found. Lather, rinse, repeat.

Try It Now

Amazon::API and Amazon::API::Help are available from CPAN. Generated AWS service distributions are available from the Amazon::API DarkPAN (https://cpan.openbedrock.net/orepan2). See https://cpan.openbedrock.net/signature for details on verifying the authenticity of Amazon::API service classes.

The easiest way to try Amazon::API 2.8.0 is with STS and GetCallerIdentity.

For example, using cpm:

cpm install -L local Amazon::API
cpm install -L local Amazon::API::Help
cpm install -L local --resolver 02packages,https://cpan.openbedrock.net/orepan2 Amazon::API::STS
export PERL5LIB=$(pwd)/local/lib/perl5:$PERL5LIB
export PATH=$(pwd)/local/bin:$PATH

Once installed, Amazon::API::Help can show you the operations available for a service:

amzn-api-help sts

You can then inspect an individual operation:

amzn-api-help sts GetCallerIdentity

With AWS credentials available in the environment or by letting Amazon::Credentials discover them for you, making the call is straightforward:

perl -MAmazon::API::STS -MData::Dumper \
  -e 'print Dumper(Amazon::API::STS->new->GetCallerIdentity);'

The result is returned as ordinary Perl data.

From there, install another generated service distribution and use amzn-api-help to discover its operations and request and response shapes.

Try it using Docker

If you just want to try Amazon::API without installing anything locally, a pre-built Docker image is available from Docker Hub. Assuming you have AWS credentials that can be discovered from your AWS CLI configuration in $HOME/.aws:

docker run --rm \
  -v "$HOME/.aws:/root/.aws:ro" \
  -e AWS_PROFILE=my-profile \
  rlauer/amazon-api:2.8.0 \
  perl -MAmazon::API::STS -MData::Dumper \
       -e 'print Dumper(Amazon::API::STS->new->GetCallerIdentity)'

If you would rather see exactly how the environment is assembled, you can build the environment from scratch using the bootstrap script below.

#!/usr/bin/env bash
# copy the script below as 'bootstrap' in your current directory...then:
# docker run --rm -it \
#   -v "$(pwd)/bootstrap:/bootstrap:ro" \
#   -v "$HOME/.aws:/root/.aws:ro" \
#   debian:trixie /bin/bash /bootstrap

RESOLVER="--resolver 02packages,https://cpan.openbedrock.net/orepan2"

apt-get update && \
apt-get install -y --no-install-recommends \
    perl libssl3 libexpat1 zlib1g ca-certificates gcc make  \
    libssl-dev libexpat-dev zlib1g-dev libperl-dev curl && \
    curl -fsSL https://raw.githubusercontent.com/skaji/cpm/main/cpm \
        | perl - install -g App::cpm && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

cpm install -L /app/local $RESOLVER Amazon::API::STS

# Substitute your profile below
AWS_PROFILE=sandbox \
perl -I /app/local/lib/perl5 -MAmazon::API::STS -MData::Dumper \
     -e 'print Dumper(Amazon::API::STS->new->GetCallerIdentity)';

…then from the command line:

docker run --rm -it \
  -v "$(pwd)/bootstrap:/bootstrap:ro" \
  -v "$HOME/.aws:/root/.aws:ro" \
  debian:trixie /bin/bash /bootstrap
 ...
 ...
 ...
$VAR1 = {
          'Arn' => 'arn:aws:iam::123456789012:user/sandbox-developer',
          'Account' => '123456789012',
          'UserId' => 'AIDA*************MTGW'
        };

Links


Previous post: Amazon::S3 v2.1.0 - On Being an S3 Client