AppRunner - The Good, Bad and the Ugly
Background
In May of 2021, AWS released AppRunner to the public.
AWS App Runner is an AWS service that provides a fast, simple, and cost-effective way to deploy from source code or a container image directly to a scalable and secure web application in the AWS Cloud. You don’t need to learn new technologies, decide which compute service to use, or know how to provision and configure AWS resources.
App Runner connects directly to your code or image repository. It provides an automatic integration and delivery pipeline with fully managed operations, high performance, scalability, and security.
What makes AppRunner so compelling are these important features:
- Creates the entire infrastructure required for hosting a scalable web application
- Connects directly to your code or image repository AND can automatically redeploy the application when assets are updated
- Can scale up or down based on traffic
- Very cost effective - you only pay for resources consumed when your application is in use
- Associate your custom domain with an AppRunner endpoint.
- It’s much less expensive than provisioning all of the resources necessary to host your application yourself!
My Use Case
Back in 2012, I started a SaaS application (Treasurer’s Briefcase) for providing recordkeeping services for small non-profit organizations like PTOs, PTAs and Booster clubs. Back then, I cobbled together the infrastructure using the console, then started to explore CloudFormation and eventually re-architected everything using Terraform.
The application is essentially based on a LAMP stack - well sort of since I use a different templating web framework rather than PHP. The stack consists of an EC2 that hosts the Apache server, an EC2 that hosts some backend services, an ALB, a MySQL RDS instance and a VPC. There are a few other AWS services used like S3, SQS and EFS, but essentially the stack is relatively simple. Even so, provisioning all of that infrastructure using Terraform alone and creating a development, test, and production environments was a bit daunting but a great learning experience.
Starting with the original infrastructure, I reverse engineered it
using terraforming and then
expanded it using terraform.
The point being, it wasn’t necessarily easy to get it all right the first time. Keeping up with Terraform was also a challenge as it evolved over the years too. Moreover, maintaining infrastructure was just another task that provided no incremental value to the application. Time spent on that task took away from creating new features and enhancements that could provide more value to customers.
Enter AppRunner…with the promise of taking all of that work and chucking it out the window. Imagine creating a Docker container with your application and handing it to AWS and saying “host this for me, make it scalable, create and maintain an SSL certificate for me, create a CI/CD pipeline to redeploy the application when I make changes and make it cheap.” I’m in.
Not So Fast Skippy
AppRunner has evolved over the years and has become much more mature. However, it still has some warts and pimples that might make you think twice about using it. Back in 2021 it was an interesting new service, an obvious evolutionary step from Fargate Tasks which provide some of the same features as AppRunner. Applications that utilized Fargate Tasks as the basis for running their containerized web applications still had to provision a VPC, load balancers, and manage scaling on their own. AppRunner bundles all of those capabilities and creates a compelling argument for moving Fargate based apps to AppRunner.
Prior to October 2022 AppRunner did not support the ability to access resources from within a VPC. That made it impossible for example, to use a non-publicly accessible RDS instance. With that addition in October of 2022, it was now possible to have a web application that could access your RDS in your VPC.
The fall of 2023 has seen several changes that make AppRunner even more compelling:
| Change | Description | Date |
|---|---|---|
| Release: App Runner adds supports for AWS Europe (Paris), AWS Europe (London), and AWS Asia Pacific (Mumbai) Regions | AWS App Runner now supports AWS Europe (Paris), AWS Europe (London), and AWS Asia Pacific (Mumbai) Regions. | November 8, 2023 |
| Release: App Runner adds dual stack support for incoming network traffic | AWS App Runner now adds dual stack support for incoming traffic through public endpoints. | November 2, 2023 |
| Release: App Runner automates Route 53 domain configuration for your services | AWS App Runner automates Route 53 domain configuration for your App Runner service web applications. | October 4, 2023 |
| Release: App Runner adds support for monorepo source-code based services | AWS App Runner now supports the deployment and maintenance for monorepo source-code based services. | September 26, 2023 |
| Release: App Runner adds more features to auto scaling configuration management | AWS App Runner enhances auto scaling configuration management features. | September 22, 2023 |
Some of the limitations of AppRunner currently include:
- Inability to mount file systems (like EFS)
- While you can associate a custom domain, it cannot provision www from the console (in fact I have been unable to get it to work properly from the CLI either although the documentation indicates it should work)
- Still unavailable in some regions
- Cannot use security groups to limit access
The first limitation is a bit of show-stopper for more than a few web applications that might rely on mounted file systems to access assets or provide a stateful storage environment. For my application I use EFS to create session directories for logged in users. Using EFS I can be assured that each EC2 in my web farm accesses the user’s session regardless of which EC2 serves the request. Without EFS, I will be forced to re-think how to create a stateful storage environment for my web app. I could use S3 as storage (and probably should) but EFS provided a “quick-shot-on-goal” at the time.
The second limitation was just frustrating as associating a custom domain sort of kinda works. If I associate a domain managed by AWS (in the same account as my AppRunner application) then I was able to get the TLD to resolve and work as expected. AppRunner was able to associate my appliation to the domain AND provide an SSL certificate. It will redirect any http request to https. Unfortunately, I could not associate www sub-domain using the CLI as documented. In fact I could not even get the CLI to work without trying to enable the www sub-domain. Working with AWS support confirmed my experience and I still have a ticket pending with support on this issue. I’m confident that will be resolved soon(?) so it should not limit my ability to use this service in the future.
Conclusion
AppRunner is an exciting new service that will make application development and deployment seamless allowing developers to focus on the application not the infrastructure.
You can find the AppRunner roadmap and current issues here.
I’m So Done With Agile
Every development project ultimately has a goal of providing some kind of value to the organization that has decided to initiate a software development project.
The bottom line of any software development project is the bottom line. Does the cost of the project AND the maintenance of the project create a profit?
I know what you are thinking. Not all software applications are designed to produce profit. Untrue. Even applications we call “internal” create value or contribute to the creation of value.
What is Failure?
Let’s talk about and characterize failure first. Because its much easier to define (as anyone who has had the misfortune of working with a product development team that cannot define “done” knows). And I’ve been told that that most software development projects fail.
The project is canceled.
This is the “first order broke” condition of projects. It took too long, it went over budget and looked to continue to be a money pit (someone understood the fallacy of sunk costs), the environment changed making the application moot or a new CEO decided to replace all internal applications with some SaaS, PaaS, or his own pet project.
The application was launched and did not meet the goals of the project.
This can mean a lot of things: the project does not solve enough of the business problems to justify the continued cost of maintenance. Or perhaps the application did not generate enough revenue to justify its existence because of poor market acceptance. People just hate using it.
The project is in use, people use it, but the ROI is too far in the future or perhaps indeterminate.
The project becomes a drag on the organization. No one wants to pull the plug because they have no alternative (or believe they don’t). There’s no appetite to rewrite, refactor or reimagine the application. It becomes a huge boat anchor that a handful of engineers keep running by kicking it in the ass whenever it stalls.
What is Success?
The project launches on time and under budget.
Keep in mind that this is (mostly) a necesasry, but insufficient condition for success. Yes, there are some successful projects that are over budget or late, but its sort of like starting Monopoly owing everyone money. You need to catch up and catch up fast.
The application completely solves the business problem.
Again, a necessary but insufficient condition for success. If the application is difficult to maintain and requires constant attention that costs more than it saves or produces, it’s not a success.
The application just works
…and is a critical component in a complex workflow - without it nothing else would - its cost to develop and maintain is easily justified by the the nature of its job. It successfully completes its mission every single day.
This Was About Agile Right?
Oh yeah, Agile. I read articles about Agile and people’s experience with it all the time. I suspect most opinions are based on few data points and mostly from one person’s negative (or rarely positive) experience with Agile. My opinions (and that’s all they are…YMMV) are based on working with some fairly large clients that I am not at liberty to divulge. One FANG, one Fortune 50 company, one major manufacturer of phones and multiple companies with more than 5000 employees. I’m not opining based on one ride on the merry-go-round. I’m the kind of person that always believes that I just don’t get it, and I need to learn more, read more and accept more to overcome my ignorance and lack of experience. It’s a viewpoint that has allowed me to grow in my career and learn a lot of very useful things that have conspired to make me, if not wealthy, not concerned about money.
I am now having a lot fun going back to my roots of being a software developer. While I have been on the management side of projects employing the Agile process I am now in the belly of the beast. It smells bad, feels wrong and kills productivity. But, again, YMMV.
Why Does Agile Suck?
Product Owners - All “product owners” are not created equal. They have varying degrees of understanding of their own domain. Some even believe developers have ESP. To be fair, some expect developers (and rightly so) to “ask questions”. The problem is, what happens when the developer does not understand the domain. What questions should they ask? They are clueless.
Product owners should assume nothing (in my opinion) and determine the level of domain expertise developers have. It is their responsibility to make that assessment - if they don’t they must be explicit with requirements, otherwise you’ll almost certainly end up with a project or feature that does not meet your needs.
- Scrum Masters - are generally useless. The Vanna White’s of software development. I’d almos say Vanna had a harder job. Turning letters that have been lit up for you puts a lot of pressure on you to turn the “right” letter.
- Ceremonies - Most developers hate meetings. There is of course the rare “meeting moth” that has decided it is easier to pontificate, hash things out, and “parking lot” ideas than actually code. But let’s talk about the programmers we all know and love. Leave them alone with a problem and some good requirements, decent tools and they’ll produce something for you. Developers generally loathe people. To hold meetings apparently you need to round some people up and torture them for 30 to 45 minutes every day. Oh, so let’s make “stand-ups”! We’ll cap them at 15 minutes and 20 people will get a fraction of a minute to recap what they did yesterday. Only there’s that one guy. Yeah, you know who you are. You think we care about your busy day and the fact that you worked through the night solving problems you created? Nope. Please, please, continue babbling so the time runs out on this farce!
- Pointing - Sadly, there’s a poor Italian mathematician turning over in his grave as we speak. His claim to fame usurped by idiots who use numbers without understanding what they represent or the context for their use. That’s right - points mean diddlysquat. I’ve seen them used for billing by the scammy outsourcing companies that point their own stories and then bill accordingly. Or there is the pointing that uses them to determine how much “work” someone can do in 2 weeks. Ha! Gotcha! I’m pointing everying as 13! Is that Fibonacci?
- Retrospectives - Fuck that shit. Let’s just do the Festivis pole, air our list of grievances and then proceed to the feats of strength. It would be more useful and of course highly entertaining. Hoochie Mama!
- Sprints - We don’t know where we’re going but let’s sprint there! Isn’t that the irony of Agile? We don’t know wtf the requirements are until we start coding (according to some people’s definition of Agile) and yet we want to get it done quickly. It’s like watching my 3 year old son on his Big Wheel motoring down the street careening off of curbs and smashing this absolute wonder of plastic technology into my neighbors bushes. No one is seriously hurt, it’s a bit amusing but we really haven’t gotten anywhere.
So, here’s the bottom line. Any idea worth something greater than 0 that also has a wee bit of marketing behind it quickly becomes an opportunity for gypsies, tramps and thieves to exploit the ignorant masses. Take Christianity for example. Need I say more? Agile has become the Chrisitianity of corporate America. No one dare mention that is doesn’t solve our problems or make us feel any better. Fuck Agile, the ceremonies, the training, the roles the practice…it is the most unproductive enviroment one can devise for developing software. Look it up…Bill Gates wrote an entire BASIC interpreter and shoved it into 4K of a ROM. He then worked on a 32K version that was essentially a complete OS. He didn’t need Agile to do that.
So, let’s be clear. Agile is social engineering. An attempt to organize human beings in order to create something that no one of them could do alone (or so it goes). Somehow I don’t think Agile works. Some will say, yeah, well not every project should use Agile. Yes, that’s true, but the sad fact is that corporate America is not nuanced. They are binary. They want single solutions to complex problems and do not want to hear…it depends. And so they consume the entire bottle of aspirin.
There will be a day when people look back at the unproductive, waste and utter insansity that is “Agile”. They will marvel at the way that a single, possibly good idea for some things, was transformed into a dogma that haunted software development for a decade.
I’m hopeful however that really smart companies know that instituting things like Agile are the bellweather of their demise. They will avoid trying to fit round pegs into square holes. They will embrace the idea that you can plan things properly, but plans can change without embracing a chaotic, highly disorganized process that actually masquerades as a structured protocol.
You have been warned. When some consultant you hire to justify the outsourcing of your development team says that they can replace your current processes with an Agile team from Elbonia and a scrum master from Bumblefuck…be afraid…be very afraid. There is no free lunch.
One final thought…why is software development so hard? And why do we struggle so to create applications?
It’s not a hard question actually. The goal of software development is to codify a solution to a problem. But first…and here is the reveal…you have to define the problem. That is, in and of itself the most difficult thing in the development process. Missed requirements are, in my experience, the biggest reason for “re-work”. Note I did not say “bugs” or “defects”. Most maintenance on systems is because of missed requirements, not because programmers make mistakes. Oh, for sure, they do. But really? Think. Look back at your tickets and do a root cause analysis.
There are other reasons software development is hard. First, people do not communicate well. The do not communicate precisely and they do not communicate accurately. Next, the tools to express the solutions to our problems are complex and incomplete. Better ingredients make better pizzas. Papa Johns!
Okay, I have to wrap this up…Agile sucks. I hate Agile. I want to mute myself when I’m in stand-ups just to say every day “Oh, I was on mute.” and torture everyone that thinks this ceremony is useful.
Oh,I’m having issues with my internet so I may have to drop soon….open the pod bay doors Hal?
Flavio Poletti recently blogged about
using Barcode::ZBar for reading QR codes. In a previous life I have
written applications for workflows that involved reading QR codes from
documents and have also used Barcode::ZBar very effectively…highly
recommended.
Back then, iirc I had quite a struggle satisfying all of the dependencies required to get it actually work. I spent a little time after reading his article to see I could successfully build it once more, this time in a Docker container. The result is this project:
It was a little difficult piecing together the required packages, but in the end it works!
Next, I may create a webservice from the container and install it on a
Fargate cluster. The service will allow you to upload .png files and
scan them for QR codes. Stay tuned!
Other Links of Interest
Session #1 - refactoring a block containing map
The code below violates more than a few best practices making the intent of the code difficult to understand. Let’s get started…
User
Can you refactor this snippet of Perl code?
if (@multi_elems) {
my $delim_qr1 = qr/\d+$group_delim1/o;
#my $delim_qr2 = $group_delim2;
foreach my $entry ( @log_data{@EIDs} ) {
@{$entry}{@multi_elems}
= map {
my $x = $_;
$x && ( $x =~ s/$delim_qr1//go )
? [
# XXX: if $_ =~ m/^($group_delim2)+$/
# the result is ()
# at least on perl 5.8.0
split /$group_delim2/, $x
]
: []
} @{$entry}{@multi_elems};
}
#print STDERR "$err\n" if $err;
}
ChatGPT
Here’s the refactored version of your provided Perl code. I’ve added comments to explain what the code is doing and made some minor improvements for readability:
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/o;
# Iterate through each entry in the log_data hash
foreach my $entry (@log_data{@EIDs}) {
# Update each multi_elem in the current entry
@{$entry}{@multi_elems} = map {
my $x = $_;
# Check if the value matches the delimiter regex
$x && ($x =~ s/$delim_qr1//go) ?
[
# Split the value by group_delim2
split /$group_delim2/, $x
]
: [] # If the value doesn't match, return an empty array reference
} @{$entry}{@multi_elems};
}
}
The refactored code is more readable and has comments explaining the purpose of each block. Note that the code logic has not been changed, and it should function the same as the original snippet.
User
The use of the ternary operator obfuscates the meaning
ChatGPT
I understand your concern. Here’s the refactored version of the code
using an if statement instead of the ternary operator to improve
readability:
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/o;
# Iterate through each entry in the log_data hash
foreach my $entry (@log_data{@EIDs}) {
# Update each multi_elem in the current entry
@{$entry}{@multi_elems} = map {
my $x = $_;
my $result;
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//go)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
} else {
# If the value doesn't match, return an empty array reference
$result = [];
}
$result;
} @{$entry}{@multi_elems};
}
}
In this version, I replaced the ternary operator with an if-else
statement, which assigns the result to the $result variable. This
should make the code more understandable and easier to follow.
User
Excellent but I don’t like cuddled elses
ChatGPT
Sure, I’ve updated the code to uncuddle the else statement:
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/o;
# Iterate through each entry in the log_data hash
foreach my $entry (@log_data{@EIDs}) {
# Update each multi_elem in the current entry
@{$entry}{@multi_elems} = map {
my $x = $_;
my $result;
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//go)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
}
else {
# If the value doesn't match, return an empty array reference
$result = [];
}
$result;
} @{$entry}{@multi_elems};
}
}
Now, the else statement is on a new line, making the code more
aligned with your preferred style.
User
We are using Perl 5.16. Are there any changes you might suggest?
ChatGPT
Perl 5.16 doesn’t introduce any significant changes that would impact the provided code snippet. However, I can suggest a few general improvements for better readability and maintainability:
- Use strict and warnings pragmas to enforce stricter code quality:
use strict; use warnings; - Use the
foreachloop with a lexical iterator variable to avoid potential conflicts with global variables:for my $entry (@log_data{@EIDs}) { # ... }
With these suggestions, the updated code looks like this:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/o;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Update each multi_elem in the current entry
@{$entry}{@multi_elems} = map {
my $x = $_;
my $result;
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//go)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
}
else {
# If the value doesn't match, return an empty array reference
$result = [];
}
$result;
} @{$entry}{@multi_elems};
}
}
These changes should enhance the code quality and maintainability while keeping the same functionality.
User
My understanding is that the ‘o’ regex modifier may no longer be required.
ChatGPT
You’re correct. The /o modifier for regular expressions is no longer
needed in most cases, as Perl optimizes the regex automatically if the
pattern doesn’t change during the runtime of the program. The /o
modifier was useful in older versions of Perl (prior to 5.6) when the
regex engine didn’t optimize regexes in the same way. In modern Perl,
the engine is smart enough to recognize when a regex is constant and
cache the compiled version automatically. You can remove the /o
modifier from the regex in your code snippet:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Update each multi_elem in the current entry
@{$entry}{@multi_elems} = map {
my $x = $_;
my $result;
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//g)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
}
else {
# If the value doesn't match, return an empty array reference
$result = [];
}
$result;
} @{$entry}{@multi_elems};
}
}
The performance and behavior of the code should remain the same after
removing the /o modifier.
User
Perl best practices indicates that a map function should only contain one statement. Using the map function above seems to reduce readability since the object being interated over with some complex logic is at the end of the expression.
ChatGPT
I agree with you. Using a map function with complex logic can reduce
readability. In this case, we can refactor the code to use a for
loop instead of the map function to improve readability:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Iterate through each multi_elem in the current entry
for my $multi_elem (@multi_elems) {
my $x = $entry->{$multi_elem};
my $result;
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//g)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
}
else {
# If the value doesn't match, return an empty array reference
$result = [];
}
# Update the multi_elem in the current entry
$entry->{$multi_elem} = $result;
}
}
}
By using a for loop, the code becomes more readable, and it’s easier
to understand the logic being applied to the elements.
User
I think the else statement in the inner for loop can be removed and we
can assign result to an empty array reference when we define $result
ChatGPT
You’re right. We can simplify the code further by initializing
$result to an empty array reference and updating it only when the
condition is met. Here’s the updated code:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Iterate through each multi_elem in the current entry
for my $multi_elem (@multi_elems) {
my $x = $entry->{$multi_elem};
# Initialize the result to an empty array reference
my $result = [];
# Check if the value matches the delimiter regex
if ($x && ($x =~ s/$delim_qr1//g)) {
# Split the value by group_delim2
$result = [split /$group_delim2/, $x];
}
# Update the multi_elem in the current entry
$entry->{$multi_elem} = $result;
}
}
}
By initializing $result to an empty array reference, we can remove
the else statement, making the code more concise and easier to read.
User
The $result variable also now seems unnecessary and we can
further refactor this by directly assigning $entry->{$multi_elem} to
an empty array and using the next statement to short circuit the if
statement
ChatGPT
You’re right. We can further simplify the code by removing the
$result variable and directly assigning the value to
$entry->{$multi_elem}. Here’s the refactored code:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Iterate through each multi_elem in the current entry
for my $multi_elem (@multi_elems) {
my $x = $entry->{$multi_elem};
# Initialize the multi_elem in the current entry to an empty array reference
$entry->{$multi_elem} = [];
# Check if the value matches the delimiter regex, and short circuit if not
next unless ($x && ($x =~ s/$delim_qr1//g));
# Update the multi_elem in the current entry by splitting the value by group_delim2
$entry->{$multi_elem} = [split /$group_delim2/, $x];
}
}
}
By directly assigning an empty array reference to
$entry->{$multi_elem} and using the next statement to short
circuit the loop if the condition is not met, the code becomes even
more concise and easier to read.
User
perl best practices suggests avoiding unless
ChatGPT
You’re right. Using unless can sometimes make the code less
readable. Instead, we can use an if statement with a negated
condition. Here’s the updated code:
use strict;
use warnings;
# Check if there are any multi_elems
if (@multi_elems) {
# Define the regex delimiter for group_delim1
my $delim_qr1 = qr/\d+$group_delim1/;
# Iterate through each entry in the log_data hash
for my $entry (@log_data{@EIDs}) {
# Iterate through each multi_elem in the current entry
for my $multi_elem (@multi_elems) {
my $x = $entry->{$multi_elem};
# Initialize the multi_elem in the current entry to an empty array reference
$entry->{$multi_elem} = [];
# Check if the value matches the delimiter regex, and short circuit if not
if (!($x && ($x =~ s/$delim_qr1//g))) {
next;
}
# Update the multi_elem in the current entry by splitting the value by group_delim2
$entry->{$multi_elem} = [split /$group_delim2/, $x];
}
}
}
By using an if statement with a negated condition instead of
unless, the code conforms better to Perl best practices and is
easier to understand.
User
How do you think this code would compare to the original in terms of performance under perl 5.16
ChatGPT
The refactored code should have similar performance to the original
code under Perl 5.16. The changes made in the refactored version
mostly deal with readability and best practices, but the overall logic
and flow of the code remain the same. Some improvements in the
refactored code, such as removing the unnecessary else statement and
the $result variable, can slightly reduce memory usage and improve
the code’s readability without affecting the performance. The direct
assignment of an empty array reference and the use of the next
statement can make the code more efficient in some cases by avoiding
unnecessary assignments. However, it’s important to note that the
actual performance difference might not be significant, and it would
depend on the specific use case, data being processed, and the overall
complexity of the program. In general, focusing on readability and
maintainability is a good approach, as it will make it easier for you
and others to work with and optimize the code in the future.
Session 2 - Refactoring Obfuscated Code
Yikes! What the heck does this code do?
User
Please refactor this Perl snippet for readability
@{$_}{ @elem_ids_ref, @elem_ids_noref }
= ( ( map { [] } @elem_ids_ref ), @defaults_noref )
for @log_data{@EIDs};
ChatGPT
Here’s the refactored version of the provided Perl snippet with improved readability:
for my $entry (@log_data{@EIDs}) {
# Initialize array references for elements in @elem_ids_ref
@{$entry}{@elem_ids_ref} = map { [] } @elem_ids_ref;
# Assign the default values for elements in @elem_ids_noref
@{$entry}{@elem_ids_noref} = @defaults_noref;
}
In this refactored version, I’ve replaced the single line of code with
a more readable for loop that iterates over each entry in
@log_data{@EIDs}. Within the loop, I’ve separated the assignment of
array references for elements in @elem_ids_ref and the assignment of
default values for elements in @elem_ids_noref. This should make the
code easier to understand and maintain.
Session 3 - Creating a Rules Engine from a Snippet
In this session I have an inkling what I want to do with this code, remove the
nextstatements and make this more of a condition based dispatcher.
User
The code below uses the next statement 4 times and uses unless which is not considered to be a best practice. How could we avoid using next in this fashion?
foreach my $elem ( $self->elements() ) {
next unless $elem->table();
next
if ( $elem->{type} eq 'File' || $elem->{type} eq 'Signature' )
&& !$elem->copy_file;
next unless $elem->$allow_hybrid_method;
next unless exists $data->{ $elem->{uniq} };
$elem->{value} = $data->{ $elem->{uniq} };
}
ChatGPT
Here’s a refactored version of the provided Perl code snippet that
avoids using next and unless:
foreach my $elem ($self->elements()) {
if ($elem->table()
&& !($elem->{type} eq 'File' || $elem->{type} eq 'Signature') || $elem->copy_file
&& $elem->$allow_hybrid_method
&& exists $data->{$elem->{uniq}}
) {
$elem->{value} = $data->{$elem->{uniq}};
}
}
In this version, I combined all the conditions into a single if
statement, which makes the code more straightforward and easier to
read.
User
That does seem more readable - each of the conditions is essentially a rule that results in assigning a value to a hash member - is there another way to express this functionality?
ChatGPT
Yes, you can create an array of coderefs representing each condition, and then iterate through them to check whether all conditions are met. This approach makes the code more modular and easier to extend or modify in the future:
my @conditions = (
sub { $_[0]->table() },
sub { $_[0]->{type} ne 'File' && $_[0]->{type} ne 'Signature' || $_[0]->copy_file },
sub { $_[0]->$allow_hybrid_method },
sub { exists $_[0]->{uniq} && exists $data->{ $_[0]->{uniq} } },
);
foreach my $elem ($self->elements()) {
my $all_conditions_met = 1;
for my $condition (@conditions) {
unless ($condition->($elem)) {
$all_conditions_met = 0;
last;
}
}
if ($all_conditions_met) {
$elem->{value} = $data->{$elem->{uniq}};
}
}
In this version, we define an array of coderefs (@conditions) that represent each condition. Then, during the iteration over elements, we check whether all conditions are met by iterating through the @conditions array. If all conditions are met, we assign the value to the hash member. This approach makes it easy to add, remove, or modify conditions in the future.
User
Here we go with unless again
ChatGPT
Apologies for that oversight. Let’s rewrite the code snippet without
using unless:
my @conditions = (
sub { $_[0]->table() },
sub { $_[0]->{type} ne 'File' && $_[0]->{type} ne 'Signature' || $_[0]->copy_file },
sub { $_[0]->$allow_hybrid_method },
sub { exists $_[0]->{uniq} && exists $data->{ $_[0]->{uniq} } },
);
foreach my $elem ($self->elements()) {
my $all_conditions_met = 1;
for my $condition (@conditions) {
if (!$condition->($elem)) {
$all_conditions_met = 0;
last;
}
}
if ($all_conditions_met) {
$elem->{value} = $data->{$elem->{uniq}};
}
}
Now, the unless keyword has been replaced with a negated if
statement, which makes the code more in line with the best practices.
User
The foreach loop looks very generic and could be a subroutine of its own, with an argument of the conditions
ChatGPT
You’re right, the loop can be extracted into a separate
subroutine. Here’s a refactored version of the code, where the loop is
wrapped in a subroutine called process_elements:
sub process_elements {
my ($elements, $data, $conditions) = @_;
ELEMENT:
foreach my $elem (@$elements) {
for my $condition (@$conditions) {
next ELEMENT unless $condition->($elem);
}
$elem->{value} = $data->{$elem->{uniq}};
}
}
my @conditions = (
sub { $_[0]->table() },
sub { $_[0]->{type} ne 'File' && $_[0]->{type} ne 'Signature' || $_[0]->copy_file },
sub { $_[0]->$allow_hybrid_method },
sub { exists $_[0]->{uniq} && exists $data->{ $_[0]->{uniq} } },
);
process_elements([$self->elements()], $data, \@conditions);
The process_elements subroutine takes a list of elements, data, and
an array reference of conditions as its arguments. It iterates through
the elements, and for each element, it checks if all conditions are
met. If all conditions are satisfied, it assigns a value to the
element’s hash member.
User
The calling sequence of process_elements() is specific to this use
case. How can we make it more generic?
ChatGPT
To make the process_elements subroutine more generic, you can pass a
code reference for the action to be performed when all conditions are
met. This way, the subroutine can be used in various scenarios, not
just for the specific use case. Here’s the modified version:
sub process_elements {
my ($elements, $conditions, $action) = @_;
ELEMENT:
foreach my $elem (@$elements) {
for my $condition (@$conditions) {
next ELEMENT unless $condition->($elem);
}
$action->($elem);
}
}
my @conditions = (
sub { $_[0]->table() },
sub { $_[0]->{type} ne 'File' && $_[0]->{type} ne 'Signature' || $_[0]->copy_file },
sub { $_[0]->$allow_hybrid_method },
sub { exists $_[0]->{uniq} && exists $data->{ $_[0]->{uniq} } },
);
my $action = sub {
my ($elem) = @_;
$elem->{value} = $data->{$elem->{uniq}};
};
process_elements([$self->elements()], \@conditions, $action);
Now the process_elements subroutine takes an additional argument,
$action, which is a code reference for the action to be performed
when all conditions are met. This makes the subroutine more generic
and reusable in different situations.
User
Let’s refactor process_elements() to receive a hash of arguments.
ChatGPT
To refactor the process_elements subroutine to receive a hash of
arguments, you can make the following changes:
sub process_elements {
my %args = @_;
my $elements = $args{elements};
my $conditions = $args{conditions};
my $action = $args{action};
ELEMENT:
foreach my $elem (@$elements) {
for my $condition (@$conditions) {
next ELEMENT unless $condition->($elem);
}
$action->($elem);
}
}
my @conditions = (
sub { $_[0]->table() },
sub { $_[0]->{type} ne 'File' && $_[0]->{type} ne 'Signature' || $_[0]->copy_file },
sub { $_[0]->$allow_hybrid_method },
sub { exists $_[0]->{uniq} && exists $data->{ $_[0]->{uniq} } },
);
my $action = sub {
my ($elem) = @_;
$elem->{value} = $data->{$elem->{uniq}};
};
process_elements(
elements => [$self->elements()],
conditions => \@conditions,
action => $action
);
Now the process_elements subroutine accepts a hash of arguments
instead of a list, which makes it easier to understand the purpose of
each argument when calling the subroutine.
User
Is there another reason this might be a better subroutine?
ChatGPT
Yes, there are several reasons why refactoring the process_elements
subroutine this way might be better:
- Flexibility: The refactored version of the subroutine is more flexible, as it allows for different sets of conditions and actions to be applied to the elements. This makes it more reusable for different use cases and easier to adapt when requirements change.
- Readability: By separating the conditions and action from the core loop, the code is easier to read and understand. This also makes it simpler to modify or extend the conditions and actions without affecting the core loop functionality.
- Maintainability: Encapsulating the functionality within a separate subroutine makes it easier to maintain and test. It also allows for better code organization and reduces the likelihood of introducing bugs when making changes to the code.
- Scalability: The refactored version of the subroutine can be easily extended to handle additional conditions or actions as needed, without having to modify the core loop. This makes the code more scalable and future-proof.
Overall, the refactored version of the process_elements subroutine is more modular, flexible, and easier to understand, making it a better design choice.
You may recognize the phrase in the title as Damian Conway’s admonition in Perl Best Practices regarding magic values in your programs. It’s good advice to use names for magic numbers and values to make your code more readable for you and those that have to maintain your code.
But what about the don’t use constant part? His argument for not
using the pragma essentially boils down to the fact that use
constant creates package scoped constant subroutines at compile time
vs the Readonly module which executes at runtime. So we are told to
use Readonly…fair enough…but Perl::Critic tells us to use
ReadonlyX because Readonly is slow and buggy.
Here’s the problem though: ReadonlyX is not exactly the same as
Readonly and many CPAN modules use Readonly. Case in point:
File::BOM.
Let’s start with a little script that uses both ReadonlyX and
File::BOM;
use strict;
use warnings;
use ReadonlyX;
use File::BOM;
1;
…let’s compile this and see what happens…
perl -wc foo.pl
Subroutine croak redefined at /home/rclauer/lib/perl5/Readonly.pm line 13.
Prototype mismatch: sub Readonly::Scalar ($;$) vs ($$) at /home/rclauer/lib/perl5/Readonly.pm line 257.
Prototype mismatch: sub Readonly::Scalar ($;$) vs ($$) at /home/rclauer/lib/perl5/Readonly.pm line 343.
Subroutine Scalar redefined at /home/rclauer/lib/perl5/Readonly.pm line 316.
Subroutine Array redefined at /home/rclauer/lib/perl5/Readonly.pm line 346.
Subroutine Hash redefined at /home/rclauer/lib/perl5/Readonly.pm line 364.
Subroutine Clone redefined at /home/rclauer/lib/perl5/Readonly.pm line 387.
Prototype mismatch: sub Readonly::Readonly (\[%@$]$) vs (\[$@%]@) at (eval 7) line 42.
Subroutine Readonly redefined at (eval 7) line 1.
foo.pl syntax OK
Hmmm…that looks bad, but I hear that Readonly and ReadonlyX
should be “drop in replacements for each other? Well then perhaps I
can use ReadonlyX by telling File::BOM that Readonly has already
been loaded…a technique I have used before when these two modules
were conflicting.
use strict;
use warnings;
BEGIN {
use Module::Loaded;
use ReadonlyX;
mark_as_loaded('Readonly');
}
use File::BOM;
1;
…let’s compile this and see if things get better…
perl -wc foo.pl
Useless use of reverse in void context at /home/rclauer/lib/perl5/File/BOM.pm line 204.
Odd number of elements in hash assignment at /home/rclauer/lib/perl5/File/BOM.pm line 187.
Compilation failed in require at foo.pl line 14.
BEGIN failed--compilation aborted at foo.pl line 14.
Nope. It appears that File::BOM makes use of features or uses
Readonly in a way that is not compatible with ReadonlyX. Luckily
my use of ReadonlyX is compatible with Readonly, so I can reverse
the technique and use Readonly and mark ReadonlyX loaded so that
other modules that use ReadonlyX will use Readonly.
use strict;
use warnings;
BEGIN {
use Module::Loaded;
use Readonly;
mark_as_loaded('ReadonlyX');
}
use File::BOM;
1;
Of course, this only works if the downstream modules that use
ReadonlyX use it in a way that is compatible with Readonly…and
we’ve just introduced the slow and buggy Readonly to our stack
which we were trying to avoid by using ReadonlyX in the first place.
But at least we got our legacy code to work. ;-)
make
Anyone that has seen any of my projects on
GitHub knows that I love me some
Makefile. I’ve been using make since the days I started
programming and I doubt I’m going to give it up any time soon.
I know the kids today are haters, but make, in all its wonky glory is a
true wonder and the
Leatherman
of GNU tools. I continually find new tricks that extend my love and
adoration for this ubiquitous tool.
Scriptlets?
My latest find is what I’ll term scriptlets. Many times
I’ll find myself creating a Makefile that requires some
non-trivial manipulation or transformation of data. In those cases
I’ll usually try a Perl one-liner and pipe some data through a Perl
command.
Let’s suppose file foo.txt depends on bar.txt and we need to replace all
instances of xyz with abc (yes, sed my be a better choice, but
let’s go with this for now…). Your make recipe might look like this:
foo.txt: bar.txt
perl -npe 's/xyz/abc/g;' $< > $@
But what happens when things get a little more complicated and the
Perl one-liner is insufficient for the task? Well, maybe it’s time to
write a little Perl script. Of course, if I do that, I’ll then need to include
that script in my project, make it executable and reference it from my
Makefile. Hmmm…is there a better way than cluttering the project
with scripts? Enter scriptlets…
Here’s a short example that removes all members of a hash where the
keys do not represent a file with an extension of .pm.
define create_json =
use JSON::PP;
my $file_list = JSON::PP->new->decode(<>);
my @pm_files = grep { /[.]pm$/ } keys %{$file_list};
my %new_file_list = map { $_ => $file_list->{$_} } @pm_files;
print JSON::PP->new->pretty->encode(\%new_file_list);
endef
export s_create_json = $(value create_json)
foo.json: bar.json
perl -0 -e "$$s_create_json" $< > $@
…and then:
make
You can read more about the GNU make features used above by
following these links.
Hope you find make as useful as I do…drop me a line if you have a
favorite make trick or tip.
Thanks for reading.
My new favorite Perl module is a very unassuming chap that makes your help and usage notes for your Perl scripts just a little bit sexier…
When you have a fairly complicated script that requires lots of help and usage information, it will probably scroll past one page of the display when you dump it to STDOUT.
Sure, your users could pipe the output through less or more
(…and we all know that less is more right?) but why make them do
that?
Enter IO::Pager…a clever
little module that’s quite easy to use. The pattern I’m using is to
use it, if it’s available…
my $token = eval {
require IO::Pager;
IO::Pager::open( *STDOUT, '|-:utf8', 'Unbuffered' );
};
print {*STDOUT} <<'END_OF_USAGE';
usage: $0 options command
Script to do blah blah blah...
Options
-------
...
END_OF_USAGE
Brian d foy introduced the Perl community to the idea of a script and module in the same file (the so-called modulino concept). Essentially, the idea is that you can create a script that provides a command line interface to your Perl module. Essentially, the implementation looks something like this:
package Foo
caller or __PACKAGE__->main();
# package guts
...
sub main {
my $foo = Foo->new;
use Getopt::Long;
# do something interesting here...
exit 0;
}
1;
This model comes in quite handy when writing scripts or just exercising your module during it’s creation.
I sometimes find it useful to then create a bash script that invokes
my modulino. Since I might want to point to a development version
of the script, I might set my PERL5LIB path to my local development
environment. Running the modulino script, I would then expect it to
use the version in my path. Hence the following modulino script.
#!/usr/bin/env bash
# -*- mode: sh; -*-
MODULINO="Amazon::Credentials"
MODULINO_PATH="${MODULINO//::/\/}.pm"
MODULINO_RUN=$(perl -M$MODULINO -e 'print $INC{"'$MODULINO_PATH'"}';)
test -n "$DEBUG" && echo $$MODULINO_RUN
if test -z "$MODULINO_RUN"; then
echo "$MODULINO is not installed"
exit 1;
fi
perl $MODULINO_RUN "$@"
…and then
$ amazon-credentials.sh -h amazon-credentials.sh options Formats credentials found in env, config, SSO, role Options ------- --help, -h this --ec2, get credentials from server IAM profile --env get credentials from environment variables --profile, -p get credentials from profile in credentials configuration --role get credentials from SSO role --account use with --role, specify AWS account id $ amazon-credentials.sh --profile=test export AWS_ACCESS_KEY_ID=AKI***************** export AWS_SECRET_ACCESS_KEY=**************************************** $ aws sso login $ amazon-credentials.sh --role my-sso-role --account 01234567890
More information about modulinos can be found here:
Although I’ve been programming in Perl for over 25 years, it wasn’t until recently that I’ve had a boost in productivity as great as the one I experienced over the last year. What happened?
Projects Present Opportunities for Change
Stepping out from a more management oriented role at my former
employer, they needed a technical leader for a legacy Perl web
application migration to the cloud. Somehow I let on that I had some
Perl and AWS experience and so I was tabbed as the technical lead.
The project involved some heavy lifting of years of crufty Perl code
from an on-prem, dedicated Apache server environment to a
containerized AWS environment. Aside from the challenges of trying to
get legacy Perl code running in a Docker container the team had to
mitigate findings from the company’s security scans. This included
perlcritic findings as well as findings based on other tools that
look for potential security issues.
So, the first hurdle was whittling down the perlcritic findings and
making sure we weren’t introducing new findings.
Adding perlcritic to the Build
I’m a big fan of make and autotools so naturally our build was
based on GNU
autotools. This
allowed for a repeatable, standards based, reliable and
extensible build system that worked locally using docker-compose and
worked in our CI/CD pipeline to deploy Docker images for AWS Fargate.
To make sure that we maintained a baseline of low severity findings
from perlcritic, I added a step in our Makefile that ran
perlcritic and errored out if any severity level exceeded 2. This
prevented any code from going into the repository that would trigger
security concerns since all pushes to the repository were scanned.
Moving the Checks Further Upstream
My editor of choice has always been Emacs…let the editor wars begin!
I had already added perltidy as an extension to Emacs so that
perltidy would be run before any save. Our team standardized on a
set of perltidy settings and added the .perltidyrc file to the
project along with the .perlcriticrc that configures perlcritic
reporting.
Most editors today have language specific syntax highlighting and syntax checking built-in. Flycheck is an Emac’s plugin for syntax checking. Flycheck extensions are available for almost any language and even for things like markdown, JSON and SQL.
Syntax checking as you work in your scripts is
another way to move the process of validating your code further
upstream, but what really supercharged my development efforts was
adding perlcritic checking to Emacs. The combination of Flycheck’s
support for perltidy, perlcritic and Perl syntax checking has
helped me develop faster with less errors.
Perl Best Practices isn’t just code for “standarization”, although having some standards makes maintenance a heckuva lot easier. No, PBP can highlight potential issues in your code and prevent you from having to debug gnarly problems.
Unfortunately, it’s also a nag.
Visiting an old legacy file that someone wrote back in 2012 (before you and they were woke to PBP) is an eye opening experience. When Flycheck gives up after 1527 issues found you start to question whether you really want to edit that code!
Even code that has relatively few findings presents them with electric hilighting that eventually rattles your nerves. It also becomes a bit of an obsession to clean up old code and get down to 0 findings!
In the end tough, the result is better code. Other programmers can read and understand it and the quality will get incrementally better.
Generally speaking then, to supercharge your development efforts move code quality enforcement as far upstream as possible, starting with your editor.
In physics there is the concept of the observer effect whereby the act of observing something alters the measurement. Recently, I’ve had the pleasure of experiencing that firsthand.
The evils of tied variables are well documented and frowned upon by our
friendly(?) mentor perlcritic, but let’s suppose we want to be
clever…another frowned upon attribute of Perl scripts :-( . But let’s
forge ahead anyway…
We’ll create a class that we can use to benchmark our scripts. We want this to be easy, so we’ll create a class that allow us to name benchmarks and simply access a hash element when we want to stop and report the time…sort of like this:
#!/usr/bin/env perl
use feature 'say';
use Easy::Benchmark;
use Data::Dumper;
my $benchmark = Easy::Benchmark->new( 'begin', 'start' );
sleep 2;
say $benchmark->stop('start');
$benchmark->start('next');
sleep 1;
# print {*STDERR} Dumper $benchmark
say $benchmark->stop('next');
say $benchmark->stop;
Here’s the class that implements our easy benchmark.
package Easy::Benchmark;
use strict;
use warnings;
use Benchmark;
use constant ANON_BENCHMARK => '__anon_benchmark__';
########################################################################
sub new {
########################################################################
my ( $proto, @args ) = @_;
my $class = ref $proto || $proto;
my $self = bless {}, $class;
tie %{$self}, $class, @args;
return $self;
}
########################################################################
sub TIEHASH {
########################################################################
my ( $class, @args ) = @_;
my $self = bless {}, $class;
# start the clock on an anonymous benchmark
$self->start(ANON_BENCHMARK);
for (@args) {
# and any additional ones requested.
$self->start($_);
}
return $self;
}
########################################################################
sub FETCH {
########################################################################
my ( $self, $name ) = @_;
die "$self: Unknown benchmark named `$name'\n"
if !exists $self->{$name};
if ( !ref( $self->{$name} ) || ref $self->{$name} ne 'ARRAY' ) {
$self->{$name} = [ Benchmark->new, $self->{$name} ];
}
return timestr( timediff( @{ $self->{$name} } ) );
}
########################################################################
sub DELETE { return delete $_[0]->{ $_[1] }; }
sub FIRSTKEY { my $a = scalar CORE::keys %{ $_[0] }; each %{ $_[0] } }
sub NEXTKEY { each %{ $_[0] } }
sub STORE { $_[0]->{ $_[1] } = $_[2] }
########################################################################
########################################################################
sub start {
########################################################################
my ( $self, $name ) = @_;
die "must specify name of counter\n"
if !$name;
my $start = Benchmark->new;
$self->{$name} = $start;
return $start;
}
########################################################################
sub stop {
########################################################################
my ( $self, @args ) = @_;
if ( @args > 0 ) {
return $self->{ $args[0] };
}
else {
my $name = ANON_BENCHMARK;
my $value = $self->{$name};
delete $self->{$name};
return $value;
}
}
1;
Running our script produces the expect output…
perl -I . foo.pl
2 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
1 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
3 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Let’s remove the comment on the line that dumps our $benchmark
object…and see what happens
perl -I . foo.pl
2 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
$VAR1 = bless( {
'__anon_benchmark__' => ' 2 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'begin' => ' 2 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'next' => ' 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'start' => ' 2 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)'
}, 'Easy::Benchmark' );
0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
2 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
Hmmm…I expected our last benchmark to be 3 wall seconds…that
wasn’t what I expected. By now you probably know the problem. The
class uses the FETCH method to stop a timer. So, when Dumper
fetches the keys of the $benchmark object, the timer is restarted.
The act of observing the hash triggered an action that resulted in
upsetting our experiment.
I am embarassed to say it took me an hour or so to piece it all together since I didn’t know who (or what) was accessing the hash in an effort to provide some useful log messages.
Generalizing, any time you create a class that triggers actions when fetching values from hash, be aware of the observer effect.
Thanks for reading…
www.linkedin.com/in/robert-lauer-1590513