As always, every day brings up new challenges, and today I faced one of my greatest fears - dealing with mailing DNS records.
Up until today, I was only familiar with A Record, CNAME Record, and TXT Record. A while ago, I used the MX Record while setting up my mailbox for my domain, but I don’t quite remember its purpose, but enough about me.
Mail eXchangers, such as Gmail, Yahoo, Outlook, check the validity of incoming email messages. As part of the validation process, the Mail eXchanger queries the DNS records of the sender and evaluates the sender’s reputation.
The sender’s reputation affects how the Mail eXchanger classifies the message and whether or not it should be marked as spam before it is delivered to the receiver. In some cases, the message is blocked entirely by the Mail eXchanger, and the receiver is unaware of it.
There are many algorithms out there for evaluating an email sender’s reputation. To put it on a scale, I classified the reputation level into four levels - Very Low, Low, Standard, and Best.
Blocked completely without even bothering the receiver, here are two common error messages that may appear in the ESP’s logs if the sender’s email has sent too many emails in a short period.
421 4.7.0 [TS01] Messages from <220.127.116.11> temporarily deferred due to user complaints <18.104.22.168> ;see http://postmaster.yahoo.com/421-ts01.html
553 5.7.1 [BL21] Connections will not be accepted from 22.214.171.124, because the ip is in Spamhaus's list; see http://postmaster.yahoo.com/550-bl23.html
If there’s a need to send large volumes of email messages, it is recommended to purchase a dedicated IP with a warm-up mechanism. The subject is broadly explained in the provided link, though in short, it’s another method of identifying yourself as a trusted sender, by saying:
“I own a unique IP address because I’m confident that Mail eXchangers won’t block me. I know that it’s easy to block an IP address, so yeah, I’m that confident.”
Emails are marked as spam due to the low authenticity of the sender or previous reports of other users. But, most of the time, it’s because the content of the message is not specific enough to the receiver.
For example, Willy is a Gmail customer, and he is interested in computer science and surfing. Then, out of the blue, Willy receives an email message about a great body lotion for women. Though Willy may have subscribed to some cosmetics stores websites, if the message isn’t specific enough, like “Hi Willy,” then it might be marked as spam.
Send the relevant content to the appropriate audience to avoid getting here.
Email messages are getting to the client’s mailbox as they should, though, in some Mail eXchangers such as Gmail, messages could move to the Promotions tab, while it wasn’t intended.
“… Gmail will recommend emails based on previous user engagement, regardless of whether annotations are present.”
Email messages are getting to the client’s mailbox as the sender intended.
Some businesses may want to get on Gmail’s Promotions tab on purpose; it all depends on the sender’s needs.
Are you ready to explore the dark depths of mailing DNS records? Let’s go!
Previously, I mentioned that I’m familiar with a few DNS records, but that doesn’t mean that we’re on the same page. So line up!
|Record Type||Target||Example Key Pair Values|
|A||IP addresses||Name: virtual-machine.meirg.co.il (For example, AWS Elastic IP)|
|CNAME||Canonical name for other domain name||Name: meirg.co.il|
Value: meirg-website.pages.dev (Cloudflare Pages)
Name: www.meirg.co.il (Redirects to meirg.co.il)
|TXT||Domain validation and authenticity||Name: meirg.co.il (Google Domain Verification)|
|MX||Let's leave it as a mystery for now|
(keep your Google Search gun down!)
I need to improve the email deliverability of outgoing emails. The request first came from the marketing expert of one of my customers, and he told me “We need you for the DNS part.”
Ok, I’m up for it; let’s set up the DNS records and assist my customer in reaching out to his end-users.
But how can I make sure that I succeed? I mean, doing/learning stuff is fun, it really is, but how will I know if my changes impacted my email sender reputation? How do I know if there’s a better open/click rate for my emails?
The above questions are widespread when the need to send emails arises. That usually comes with many requirements, such as “unsubscribe mechanism”, “group contacts by segments”, “statistics of open/click”, and the list goes on, depending on your marketing manager 😉
Developing a system that can measure open/click rate and handle email/contacts management will take ages to build…
So maybe I should …
Here comes ESP into play! ESP stands for Email Service Provider, and that means there are cloud providers out there who are willing to send emails on my behalf and enable me to manage the whole email delivery system in a one-stop-shop.
We covered ESPs, moving on to getting familiar with industry standards for improving email deliverability.
NOTE: I might mention SendGrid a lot during this blog post, though I’m not biased towards any ESP; SendGrid is simply the service I’ve been using recently, so it’s easier for me to make references. Most ESPs offer a free/trial plan so you can explore their features and then decide, and I recommend evaluating at least two ESPs before picking one.
According to Google, at least four standards should be implemented (in this order)
And, I’m adding A dedicated IP to the mix.
Mail eXchangers check if the sender meets industry standards by querying its DNS record, and according to that, evaluate the sender’s reputation. The sender evaluation process includes other methods, such as looking in https://multirbl.valli.org/ to see if the source domain (eventually IP) of the sender is marked as an “official spammer”, that all depends on the Mail eXchanger. The more standards the sender meets, the higher the chances the sender’s email message will arrive at its target audience (receiver).
The illustration below is for the SPF record. Still, the same communication process is the same for all standards, where the Receiving Email Server (Mail eXchanger) validates the email sender’s authenticity and reputation.
Sfor “Sending emails”
This record allows ESPs such as SendGrid, Gmail, Mailchimp to send emails on behalf of your domain. So if my domain name is meirg.co.il and I would like to send emails with Google Workspace, I need to add Google’s SPF record to my domain meirg.co.il.
And the funny thing about an SPF Record, that it’s actually a TXT Record. Let me remind you that TXT Records are for “Domain validation and authenticity,” so it’s nothing more than adding a TXT Record and adding a value that is specific for the SPF standard.
# Google Workspace Type: TXT Name: meirg.co.il Value: v=spf1 include:_spf.google.com ~all # Sendgrid Type: TXT Name: em123.meirg.co.il Value: u12345678.wl123.sendgrid.net
Notice the weird thing?
Google Workspace has an SPF expression, while
SendGrid provides a CNAME as a value in the TXT record. My gut told me that the CNAME should eventually resolve to an SPF expression, so I checked it with Authentication @ tools.wordtothewise.com.
I tested it by navigating to
https://tools.wordtothewise.com/dns/txt/em123.meirg.co.il, which resolved in … Drums …
u12345678.wl123.sendgrid.net as a TXT record which contains the following expression.
# SendGrid's CNAME resolves to SPF expression v=spf1 ip4:126.96.36.199 -all
It is quite a wonder that eventually, it’s
ip4:188.8.131.52 and not a CNAME, as we saw in Google Workspace
include:_spf.google.com. This wonder is called a dedicated IP.
Declaring an SPF Record, or any other record that is related to mailing, usually comes with a complete guide on how to create it, check Google Workspace Basic Setup for SPF which also explains how to include multiple ESPs in a single SPF Record.
Getting back to my previous example of Google Workspace and SendGrid; Luckily, SendGrid provides a particular TXT Record, which includes a unique subdomain, em123; this makes the setup is really nice since you don’t modify existing with the fear of breaking something.
Assuming your ESPs required the TXT Record of SPF to be present in the root domain, e.g., meirg.co.il, you’ll have to edit your existing SPF expression and add additional ESPs. For example, here’s how you would write an SPF expression when using both Google Workspace and Mailgun.
# Google Workspace + Mailgun Type: TXT Name: meirg.co.il Value: v=spf1 include:_spf.google.com include:mailgun.org ~all
NOTE: Regarding which ESP you should choose (Google/SendGrid/Mailchimp/Mailgun, etc.), it depends on your needs. I use Google Workspace for receiving/sending emails on behalf of human beings who use
@meirg.co.iland SendGrid for sending automated emails with my web application
@meirg.co.il. By the way, Mailchimp is also excellent, I used it a long time ago, and I truly enjoyed it.
A TXT record that contains an SPF expression or a CNAME that is eventually resolved to an SPF expression. The SPF expression contains an authorization policy about which service can send emails on your domain’s behalf.
IMPORTANT Make sure you send emails from the same address that you’ve authenticated in the ESP (SendGrid, Google), for example, DO NOT TRY THE FOLLOWING, sending emails from [email protected]. In contrast, my authenticated domain address is meirg.co.il. That can easily happen if you’re using SendGrid - API keys to send emails since the FROM field is not protected, the API call can “endure” anything you shove it.
This one is going to be very short compared to SPF since we’ve already covered all basics,
I like to think about DKIM as the HTTPS of emails. So DKIM is a way for an email sender to increase authenticity by adding a signature to the headers of an email message.
“…DKIM adds an encrypted signature to the header of all outgoing messages. Email servers (Mail eXchangers) that get signed messages use DKIM to decrypt the message header and verify the message was not changed after it was sent.”
Here’s how the DKIM record looks for my domain meirg.co.il.
Type: TXT Name: google._domainkey.meirg.co.il Value: v=DKIM1; k=rsa; p=Ultra-Long-Key-PUBLIC-KEY-392-chars
google._domainkey was generated by Google for my domain meirg.co.il. After generating the DKIM record, I followed the instructions and authenticated my domain by adding a TXT record with the value provided by Google.
If I intend to use other ESPs, I can generate a subdomain per ESP, so for SendGrid, it’ll be s1._domainkey.
As you can see, the
*._domainkey is the critical value, as that’s the one that the Mail eXchanger is checking. So if a DNS record, such as TXT or CNAME, is named with
_domainkey, you can classify it safely as DKIM. The
*._domainkey prefix, such as
s1, is called a selector. One domain can have multiple DKIM selectors when using various ESPs.
NOTE: Curious about my public key? Dig for it!
dig TXT google._domainkey.meirg.co.il
DM“Domain Monitoring” and
D-MARClike “the man(ager)”
We covered SPF and DKIM, but how do we know that our email messages are authenticated with SPF and signed with DKIM? Is there a way to track down it down with numbers? There is.
DMARC is an email authentication standard - Meeting up this standard means your SPF and DKIM domain records are legit. How does it mean that they’re legit, you ask?
Here’s how a DMARC Record looks like
p=none is the declared policy, which is nothing. Having no policy at all is almost the same as not having a DMARC record. A proper DMARC record would be
So why am I using
p=none, you ask? Because I’m still not 100% sure that my outgoing email messages pass both SPF and DKIM by Mail eXchangers. Having
p=reject would block my messages completely, so for testing purposes, start with
p=none, and once you’re sure that all your emails pass both SPF and DKIM, set the policy to
Sounds great, but how can you check all that? It sounds tough.
rua key in the code snippet above; I added
+dmarc_agg to my email address and created a filter in Gmail, so any message coming from [email protected] is tagged as
dmarc-report. Remember that I mentioned that a daily DMARC report is sent to a “predefined email address”? that is it.
But do I want to analyze DMARC reports on my own? Yup, you guessed it, there’s an excellent service that can do that for free, and it’s called ValiMail. ValiMail gets the daily report and analyzes it by checking how many SPF and DKIM successes/failures occurred in the past 24 hours and presents the summary in a friendly dashboard.
ValiMail are partnered with many known organizations, such as Microsoft, Google, and SendGrid, so I consider them a trusted vendor.
NOTE: The silent
[email protected]in the
ruakey points to ValiMail’s mailbox.
Reminding you that DMARC has the “reject policy”, so if an email sender sets it to
p=reject, pct=100 (or
p=quarantine), ALL none qualifying emails (SPF+DKIM) will not be delivered. So, cool, who enforces it? BIMI!
Qualifying for BIMI requires registering a trademark in a known patent agency and then purchasing a Verified Mark Certificate (VMC) from DigiCert or Entrust So, meeting the BIMI standard is not that simple.
And of course, it is possible to partially meet BIMI for non-trademarked logos. However, since BIMI is still not enforced or widely recognized by Mail eXchangers, I consider it a nice-to-have, and I’ll get to it once I’ll implement it once I get enough data from the DMARC reports.
To my feeling, it’ll be easier to qualify today for BIMI than it will be in a few years. If you spot a fully certified BIMI email sender, it means it is … Genuine and can be trusted.
Finally, we got to MX; in case you haven’t noticed, I wrote “Mail eXchangers” each time I had to mention a mail exchanger. You’ve just learned that MX Record stands for Mail eXchangers that are allowed to receive emails on your behalf.
Since MX Records are for receiving emails and not sending, see my stackoverflow answer on how to configure MX Records for AWS Simple Email Service (SES) and the things that I’ve learned during the process.
To see how an MX Record looks like, check mine with mxtoolbox.com.
How do you send emails from your Gmail account if it’s only receiving messages with a Mail eXchanger?
As part of being Gmail’s client, you can send emails on behalf of
@gmail.com. That is set behind the scenes by Gmail’s engineers, so each time you send an email using Gmail’s UI, or use Gmail’s API, the emails that you send are signed with Gmail’s signature, which includes SPF, DKIM, DMARC. That is why your emails are not marked as spam when you email your friends and family since most Mail Exchangers treat
@gmail.com as a high reputation sender.
All records refer to the email sender’s domain.
Check if your DMARC record is valid with https://tools.wordtothewise.com.
Hooray for Google Workspace customers - there’s a dedicated online tool for checking whether your mailing DNS records are correctly set for Google Workspace ESP; here are my results meirg.co.il - Google Check MX Status
IMPORTANT: Do not be fooled; Google Workspace’s tool is valid only if you’re checking Google as an ESP. If you use this service on a Sendgrid domain, it will show many errors.
The syntax of how to read/write a DMARC expression can be found in RFC7489 - Domain-based Message Authentication, Reporting, and Conformance (DMARC). Another excellent source for understanding the syntax is Google’s tutorial on the DMARC record format
Let’s inspect @gmail.com’s DMARC record together
v=DMARC1; p=none; sp=quarantine; rua=mailto:[email protected]
p=none - all emails, regardless if they’re validated with SPF+DKIM, can get to the receiver
sp=quarantine - all emails sent from subdomains, e.g.
subdomain.gmail.com and are not signed with SPF+DKIM are marked as spam in the receiver’s mailbox.
sp stands for subdomain-policy and overrides
p rules for subdomains.
rua=mailto - It appears that Gmail is handling their DMARC reports internally since the reporting address belongs to @google.com. So I guess they have their own “analyzing and monitoring DMARC reports” system, kewl.
Inpsecting @yahoo.com’s DMARC record
p=reject - all emails that are not signed with SPF+DKIM are rejected.
pct=100 - the percentage of messages to apply the
p policy. In this case, if it’s not signed with SPF+DKIM, it’s not getting to the receiver at all, very restrictive
It’s fascinating to see how different enterprise-grade email senders are setting up their DMARC records. It might look like Gmail is very permissive with their
p=none policy, but I’m sure they have a dedicated backend that filters out spam messages before they arrive at their customers’ mailboxes.
Yes, yes, YES!
Eventually, an email message contains text, and this text instructs the Mail eXchanger how to validate the message’s sender, process referenced images to be shown in the UI, etc.
Gmail made it very easy to check, click the ellipsis, and then Show Original.
That results in a very long page that contains the whole email message, though Gmail made it very easy to track the important things, such as if DMARC is valid.
After reading this blog post, you’ve just learned that checking DMARC is enough since it relies on both SPF and DKIM.
Psst! Glad to see that you’ve read it all; This blog post is long enough, so here’s a great resource that can help you decide whether to purchase a dedicated IP or not - 3.4 Shared vs. Dedicated IPs, M3AAWG Sender Best Common Practices v3.0, Feb 2015.
In short, if you’re sending high volumes, hundreds of thousands of emails a month, then you should purchase a dedicated IP. I hate it when it all ends up with “it depends on your needs”, but hey, that’s true.
I admit that it was tough and felt like there’s a lot to memorize, but once you truly understand how each components works, it’s pretty nice and very not intimidating as it was. So enjoy setting up your mailing DNS records. That would be all.