I got my first email address somewhere in 1994, shortly after logging into my first BBS. This was just a year before most of the world got on the internet with Windows 95 and modems, but I continued using BBSes along with the general internet until at least '98. I wouldn't say I was hooked, but I would say that email, and networked communication has been very important to me for 31 years.
Back in the early 2000's, I wrote a Bayesian Python anti-spam proxy called PASP. I used it to pull down yahoo, hotmail, and my regular paid-for email to a local email client. I think I still have that archive on a hard drive sitting about 1 foot from my left elbow. Since then, I've worked for an email anti-spam startup and co-written a patent on POP3 email spam filtering, was for several years the maintainer of asyncore, asynchat, and smtpd in the Python standard library (I updated them all make the Python 2.x -> 3.x). More recently, I've been slowly working on a better email interface for people, but it's been slow thanks to having a wife, kids, and a full-time job.
But that's not what I want to write about today. What I do want to write about is the weirdest email bug I've had to diagnose, on September 11, 2025. My goodness, it's been a week, and it's only Thursday.
The Setup
Like many other orgs, we've decided to run our app inside of AWS Lambda, with explicit permissions to only the handful of services that the Lambda needs. Little things like RDS Postgres, ElastiCache Valkey, SNS, SES, S3, API gateway, and a NAT gateway. Super normal stuff. Of course, like modern AWS, everything is in a VPC, so we need VPC SES endpoints (among others) in every AZ, appropriate security groups, all that jazz.
DNS is set up appropriately with SPF and domainkeys, with a hard fail for emails coming from our info.* subdomain. Also pretty normal, though most people prefer soft fail as a default, I don't think any spammer should ever be able to spoof anything into an inbox, hence the hard fails.
What Is Happening?
Notice something odd?
<spf>fail</spf>
Wait, what?
Doing a reverse DNS lookup on the provided IP shows:
PTR 23.251.226.5 Amazon.com, Inc. (AS16509) e226-5.smtp-out.us-east-2.amazonses.com 15 min
At first glance, this is a simple "two parts are doing two things at two different times" bug, seemingly at the email host. The fail reports are being generated by one part of the system, but the longer daily report with passing result is being processed somewhere else, and is getting the proper domain.
But why?
Looking around on the AWS QA site hosted by Amazon, while about to post my question, I found an almost identical DMARC report record as I posted above, to a different domain / service provider, with the IP address 23.251.226.7 from 3 months ago with an answer suggesting to add an SPF record for include:email-smtp.<region>. Don't do that, you'll then allow email-smtp.<region>.anydomain.com to send and pass your spf, even if it didn't pass domain keys. And with a soft fail config, those emails will sit in spam folders pretending to be you.
Whomever has already entered your stuff like that? I'm sorry, you've got an email spam hole that was created by AWS reverse dns confusion (it makes sense, but super annoying). That sucks. Keep reading, I've got a solution.
After asking a question in a comment, I finally remembered where I saw email-smtp.<region> before... AWS console -> VPC -> PrivateLink and Lattice -> Endpoints -> [our SES prod gateway]
Digging in, I found that because we had "Private DNS names enabled", our AWS SES DNS names inside the VPC were:
vpce-<hex>-<hex>.email-smtp.<region>.vpce.amazonaws.com
vpce-<hex>-<hex>-<region>a.email-smtp.<region>.vpce.amazonaws.com
vpce-<hex>-<hex>-<region>b.email-smtp.<region>.vpce.amazonaws.com
vpce-<hex>-<hex>-<region>c.email-smtp.<region>.vpce.amazonaws.com
email-smtp.us-east-2.amazonaws.com
Note how none of them have the `amazonses.com` that the SES config says should be sufficient? Yeah, me too. And it also would explain why email-smtp.<region> as suggested would work for all of these. It's in there, but before either amazonaws.com or vpce.amazonaws.com.
But why would email processing inside AWS be getting one of those names during a reverse lookup?
Because their email processing is in AWS too. Not just in AWS, their code is running in the same AWS region, quite possibly in the same availability zone. Maybe, they're even running some SES email ingress config?
So what's actually happening?
1. we send the email via SES, and because we've set up IP4 as our address and dns record types, plus private names, plus vpc, etc., we get one of 23.251.226.5, 23.251.226.6, 23.251.226.7 (the actual VPC endpoint IPs for our region right now)
2. the destination email processing has to be able to bounce emails (and maybe forward), so will have SES access (possibly with private DNS names enabled), so when performing reverse DNS (possibly with IP4 as endpoint and dns record types), gets the *.email-smtp.<region>(.vpce)?.amazonaws.com reverse lookup result, failing with our previous include:amazonses.com config
3. The failure is counted as being spf domain related on the reverse lookup of the IP, because it doesn't match any of our include: directives, and goes into a database or queue somewhere (with the email dropped as a hard fail, due to our spf config)
4. when the dmarc daily report is run, it is being run in a batch process outside of their email server, so when performing reverse dns lookup, without access to the VPC SES endpoint (but probably with access to the server's ability to send via it's internal API, or batch process pickup, etc.), it gets *.smtp-out.amazonses.com, which then passes in the later line
5. we get confusing reports, because things are being run in different contexts
How can we fix AWS SES email sending without leaving gaping security holes, unless / until someone upstream notices?
Make sure you've got all of these includes in your spf record, for every region you need to send email from:
include:amazonses.com
include:email-smtp.<region>.amazonaws.com
include:email-smtp.<region>.vpce.amazonaws.com
So yeah, Amazon's current default recommended config for sending via SES can sometimes fail if the destination email server is running inside the same AWS region as you are, with some super common configs. I don't know if IP6 solves the problem, as it's a reverse dns problem, and the reverse dns servers probably auto-generate names, but someone else can write that blog post.
Are you getting SPF fails from specific AWS SES IP addresses? Hopefully not after adding all of the relevant regions to your SPF config.
Do I NEED to Do This?
I'm doing this because I care about delivering emails to my paying customers, despite reverse dns configs being annoyingly weird inside of AWS regions, especially when you happen to be sending email to and from the same AWS region.
I would really prefer if AWS reverse dns was spewing something like *.email-smtp.<region>.amazonses.com or *.email-smtp.<region>.vpce.amazonses.com instead, so it could match the original include: directive, but I wouldn't even know where to start bothering AWS folks to make that happen. I mean, there should have been a reason to use the amazonaws.com domain and not the amazonses.com domain, right?
For service providers utilizing AWS SES and VPCs just realizing they are blocking / rejecting spf for emails coming in like this? Goodness. I don't know. At this moment, I'm glad I'm not running an email service provider. A lot of emails not being delivered, a lot of emails being marked as spam, every company emailing yours in common AWS regions with common configs having the same issue? Oof. That's really tough, especially because I'm not running your services.
I'm glad I found a workaround, and I'm glad that email should be deliverable consistently now for me, and anyone who finds this.
PROTIP: when including relevant email records for SPF, you can skip the MX record unless you want to receive email with AWS as well. I've seen more than a few people bork their incoming email configs by including the MX record (which arguably shouldn't even be there, but that's a different conversation).
