Log::Log4perl::Appender::CloudWatchLogs is now available on CPAN.

A Log::Log4perl::Appender for CloudWatch

Although AWS already provides ways to get application output into CloudWatch Logs, there are several reasons you might need a more direct approach.

AWS Native Log Capture

• For Lambda, runtime output is sent to CloudWatch Logs automatically
• Likewise ECS can route container output there
• The CloudWatch agent you install on an EC2 instance can tail files and forward them to multiple streams

A Different Approach

Those approaches are useful, but they mostly treat logging as an infrastructure concern.

The motivation for Log::Log4perl::Appender::CloudWatchLogs was simpler: if an application is already using Log::Log4perl, why run another daemon or sidecar just to move those log messages into CloudWatch?

The appender sends Log4perl events directly to CloudWatch Logs. There is no intermediate log file to tail and no separate logging process required simply to relay log entries.

There is another practical advantage. Runtime capture of STDOUT or STDERR generally gives you whatever stream organization the execution environment chooses. An application may want more control than that.

Different Log4perl appenders can target different CloudWatch log groups or streams, allowing the application to separate operational logs, errors, audit activity, or other classes of events instead of collapsing everything into a single runtime stream.

Log::Log4perl::Appender::CloudWatchLogs

Log::Log4perl is the iconic, go-to logging system for Perl. You probably already use this in your applications. Its flexible appender model allows applications to log to a wide variety of destinations - files, databases, email, and now CloudWatch Logs.

Log::Log4perl::Appender::CloudWatchLogs is intended to behave like any other Log4perl appender you’re already familiar with. The application logs through Log4perl; the appender handles delivery to CloudWatch Logs.

A minimal configuration names a log group and a stream:

log4perl.appender.CLOUDWATCH=Log::Log4perl::Appender::CloudWatchLogs
log4perl.appender.CLOUDWATCH.group.name=/my/application
log4perl.appender.CLOUDWATCH.stream.name=application

The appender can require the group or stream to exist already, or create them when configured to do so. That keeps resource creation an explicit choice rather than a side effect of logging.

Events are buffered and sent to CloudWatch Logs in batches. The appender handles the service limits on event count, payload size, and timestamp span, so application code does not need to know anything about the PutLogEvents API.

The result is intentionally ordinary from the application’s point of view. Existing Log4perl categories, levels, layouts, and appenders continue to work as they already do; CloudWatch Logs is simply another destination.

Organize Your Logging

Direct delivery also gives you control over how your logs are organized, making monitoring and debugging easier.

With Log4perl, different appenders can target different CloudWatch destinations. An application might send general operational messages to one stream, audit activity to another, and high-value transactional events somewhere else entirely.

For example:

application/general
application/audit
application/transactions
application/errors

That separation can make searching, retention, monitoring, and access control easier because the distinction is made at the point where the application already understands the meaning of the event.

This approach also keeps the logging architecture lightweight. For small services and containers, eliminating a daemon or sidecar that exists only to relay log output means fewer processes, less configuration, and one less moving part between the application and CloudWatch Logs.

Access Your Logs with aws-logs

Sending logs to CloudWatch is only half the job. Once they are there, you still need a practical way to inspect them.

Inspired by the Python script awslogs, the distribution also includes aws-logs, a command-line application for managing your log groups and streams. Using aws-logs, you can list groups and streams, retrieve and follow log events, and perform several administrative functions.

aws-logs -h

Some Examples

List groups and streams:

aws-logs list-groups
aws-logs -g /aws/lambda/my-function list-streams

Find the most recently active stream:

aws-logs -g /aws/lambda/my-function last-stream

Retrieve recent log events:

aws-logs -g /aws/lambda/my-function -t 30m get-stream

Follow matching log streams:

aws-logs -g /aws/lambda/my-function -t 30m --follow get-stream

A More Robust Follow

Following Lambda logs is harder than it first appears. Lambda creates new log streams as execution environments come and go, so a useful follow implementation cannot simply discover the current streams once and poll them forever. A stream created after the command starts would otherwise be missed.

When --follow is enabled, aws-logs periodically calls DescribeLogStreams while it continues polling the streams it already knows about. Newly discovered streams are added to the follow set automatically.

That means a command such as:

aws-logs -g /aws/lambda/my-function -t 1h --follow get-stream

is not limited to the Lambda streams that existed when the command started. If another execution environment creates a new stream while the command is running, periodic discovery can pick it up and begin retrieving its events as well.

Active, Idle and Retired Streams

Continuously rediscovering streams is necessary for applications that may create new streams during normal operation. At the same time, a long-running follow command should avoid wasting requests on streams that have already gone quiet. To handle both cases, aws-logs keeps followed streams in three states: active, idle, and retired.

Active streams are polled normally. When a stream returns no events and its forward token stops advancing, it is considered caught up and moved to the idle set. Idle streams are still checked, but only once per discovery interval rather than on every polling pass.

If an idle stream produces new events, it becomes active again.

Lambda streams get one additional state. After a Lambda stream has remained idle for 20 minutes, it is retired from direct polling. The stream is not forgotten, though. Periodic discovery continues to rediscover its metadata, and if CloudWatch later reports activity newer than the retirement point, the stream is returned to the active set.

This keeps active Lambda streams responsive without repeatedly calling GetLogEvents against streams that are already exhausted.

Retention != Cleanup

CloudWatch Logs retention policies expire log events, but they do not remove the log stream resources that contained those events.

A log group can have a one-day retention policy but still contain log streams whose metadata remains visible for months. DescribeLogStreams will continue to return those streams, including metadata such as the stream name and last event timestamp, even after GetLogEvents returns an empty event list because the underlying events have expired.

So a retention policy reduces the amount of log data you continue to store, but it does not remove the log stream itself. That is why aws-logs includes an explicit pruning operation that allows you to remove those stale streams.

The prune-streams command removes log streams whose last event is older than a specified age.

For example:

aws-logs -g /aws/lambda/my-function prune-streams 30d

The age is expressed as a number followed by m, h, or d for minutes, hours, or days.

A dry run can be used to see which streams would be removed without deleting them:

aws-logs --dryrun -g /aws/lambda/my-function prune-streams 30d

Together, retention and pruning address two different parts of the lifecycle:

retention policy -> expires old events
prune-streams     -> removes stale stream resources

Dependencies

Log::Log4perl::Appender::CloudWatchLogs does not interface directly with CloudWatch Logs. Instead it uses an AWS service class (Amazon::API::CloudWatchLogs) which in turn uses Amazon::API.

If you visit the MetaCPAN page for Log::Log4perl::Appender::CloudWatchLogs you’ll see this notice:

IMPORTANT: This distribution depends on one or more Amazon::API service modules published outside CPAN. You must install those modules before proceeding.

Amazon::API is published on CPAN, Amazon::API::CloudWatchLogs is not.

Log::Log4perl::Appender::CloudWatchLogs is the first CPAN distribution I am releasing that treats one of those independently published service classes as an ordinary application dependency - even though you can’t install it from CPAN.

The reason for not publishing Amazon::API service classes is primarily because AWS exposes hundreds of services. Managing and keeping hundreds of independently released CPAN distributions current is not viable. You can read more of the rationale here.

Installation

The normal method of using cpan, cpanm or cpm to install Log::Log4perl::Appender::CloudWatchLogs is insufficient since one or more dependencies will not be found on CPAN.

Amazon::API::CloudWatchLogs can, however, be found on the OpenBedrock DarkPAN.

https://cpan.openbedrock.net/orepan2

Therefore, to successfully install Log::Log4perl::Appender::CloudWatchLogs, you must first install Amazon::API::CloudWatchLogs from the OpenBedrock DarkPAN.

Installing with cpm:

cpm install Amazon::API
cpm install --resolver 02packages,https://cpan.openbedrock.net/orepan2 \
  Amazon::API::CloudWatchLogs
cpm install Log::Log4perl::Appender::CloudWatchLogs

Installing with cpanm:

cpanm Amazon::API
cpanm --mirror https://cpan.openbedrock.net/orepan2 --mirror-only \
  Amazon::API::CloudWatchLogs 
cpanm Log::Log4perl::Appender::CloudWatchLogs

Details on verifying the authenticity of generated Amazon::API service classes hosted on the DarkPAN are available at:

https://cpan.openbedrock.net/signature

Try It Now

Even if you don’t need an appender to replace your existing cloud logging solution, you may still find aws-logs a useful addition to your AWS toolbox.

To make it easy to try without installing the Perl dependencies locally, a Docker image is available with aws-logs and its generated Amazon::API::CloudWatchLogs service class already installed.

Assuming your AWS credentials and configuration are available in $HOME/.aws, you can run:

docker run --rm \
  -e AWS_PROFILE \
  -e AWS_REGION \
  -v "$HOME/.aws:/root/.aws:ro" \
  rlauer/aws-logs:latest \
  aws-logs list-groups

For example, to follow a Lambda function:

docker run --rm \
  -e AWS_PROFILE \
  -e AWS_REGION \
  -v "$HOME/.aws:/root/.aws:ro" \
  rlauer/aws-logs:latest \
  aws-logs -g /aws/lambda/my-function -t 30m --follow get-stream

The container is only a convenience. aws-logs is the same command installed by the CPAN distribution.

More to Come

This is the first CPAN distribution I am releasing that depends on an independently generated Amazon::API service class. That is the model Amazon::API was designed to support: common AWS machinery in one place, generated service interfaces maintained separately, and application code free to focus on the problem it is actually trying to solve.

Log::Log4perl::Appender::CloudWatchLogs is the first application I am publishing using that model, but it will not be the last.

Links

• Log::Log4perl::Appender::CloudWatchLogs
• Amazon::API
• Amazon::API::CloudWatchLogs
• Log::Log4perl::Appender
• Announcing Amazon::API 2.8.0
• Amazon CloudWatch Logs
• aws-logs on Docker
• awslogs (Python)