Spamfighting: Bulk sending

In parts 1 and 2 of this series, I’ve explored the current best practices for authenticating outbound email, validating inbound email, and my own system configurations for such.

When not busy with my day job, I also serve on the board of our neighborhood youth basketball league as registrar and co-webmaster. As part of these roles, I maintain the email infrastructure, and send most of the announcement emails.

I had made a point of “warming up” the IP address for this server. I’ve been using this same IP for several months, and it has a returnpath.com Sender Score of 99, so I thought I’d be in the clear.

We have about 1500 parent email addresses on our announcement mailing list, with over 160 of them going to a single domain – austin.rr.com. No surprise there – Roadrunner is a very popular ISP. The problem is this: Roadrunner’s mail servers don’t appreciate when my mail server sends an email, via this mailing list, to their inbound MX mail server. Even after mailman splits the message up so there are only 10 recepients per message. The first few messages get through, the rest get put on hold (SMTP 4.x.x try again later), allowing only a trickle of messages per hour from my one IP address.

During the Austin Snowpocalypse last week, we needed to get an announcement out to our parents, that, because schools were closed, and because we rent all our court space from the area schools, our practices and games had to be cancelled for the night. I sent that note around noon. It took until 6pm before all the @austin.rr.com emails were allowed through – just in time to get notice of a cancelled 6pm event. Note – my message came from a valid SPF mail server, had a valid DKIM key attached, and the DMARC policy is “none”, so it wasn’t blocked at that level. It was blocked because my mail server’s IP address isn’t allowed to send more than a few messages to Roadrunner subscribers each hour.

Roadrunner does provide a way for you to request relaxed rate limits for your IP. I followed their process, which uses Return Path’s Feedback Loop Management service, but my request was denied, no explaination given. Perhaps they know it’s a cloud service IP, which in theory could be given to another customer at any moment. I’ll file a request with my ISP to see if they’ll sign up with ReturnPath to be responsible for their netblocks on behalf of their customers. Not sure how well that’ll go over – it could make a lot of work for the ISP mail technicians.

One other alternative is to use an outbound mail service such as Amazon Simple Email Service, Mailchimp or SendGrid, in order to get my mails out to our player’s families in a timely manner. Mailchimp appears to have the disadvantage of needing to migrate all my mailing lists to them, instead of my existing GNU Mailman setup. SendGrid has better pure SMTP integration, and with some sendmail smarttable hacking, I could probably make that work. All three involve some increased cost to us, in the months I send a lot of announcements.

Do you send bulk mail from a cloud service? How do you ensure your mails get through? Leave your comments below.

Spamfighting: mail server configuration

In part 1 of this series, I relayed a bit of my story about my use of SPF, DKIM, and DMARC to try to reduce the spam being sent as if from my personal domain, while increasing the odds that legitimate mail from my domain gets through.

In this part, I describe how these are actually implemented in my case.

First, let me describe my email setup. I have one cloud-hosted server, smtp.domsch.com, through which all authentic *@domsch.com email is sent. Senders may be either local to this server (such as postmaster@ which sends the DMARC reports to other mail servers), or may be family members who use a hosted email service (as it happens, all use GMail) as their Mail User Agent. Users make an authenticated connection to smtp.domsch.com, which then DKIM-signs the messages and sends them on toward their destination MX server. These users may also be subscribed to various mailing lists which would break (fail to get their legitimate message through to the expected receivers) if SPF policy were anything except softfail.

As such, my SPF record looks like:
@ 7200 IN TXT "v=spf1 include:_spf.google.com a mx ~all"

Outbound, user-authenticated mail from *@domsch.com should be treated differently than inbound mail. Outbound mail requires only a DKIM milter to sign each message. Messages are signed with a DKIM key, published in my DNS:
default._domainkey.domsch.com. 7200 IN TXT "v=DKIM1\; k=rsa\; s=email\; p=(some nice long hex string)"

I publish a DMARC DNS record so I can get reports back from DMARC-compliant servers.
_dmarc.domsch.com. 7200 IN TXT "v=DMARC1\; p=none\;
rua=mailto:dmarc-aggregate@domsch.com\; ruf=mailto:dmarc-forensics@domsch.com\;
adkim=r\; aspf=r\; rf=afrf "

Inbound mail to *@domsch.com should pass each message through an SPF milter which adds a Received-SPF header, a DKIM milter to check the validity of a DKIM-signed message which adds an Authentication-Results header, and the DMARC milter which decides what to do based on the results of these other two headers, and sends results to DMARC senders.

smtp.domsch.com runs CentOS 6.x, sendmail, and a variety of milters. On outbound mail, it runs opendkim. On inbound mail, it runs smf-spf, opendkim, and opendmarc, before sending it on to its final destination. My sendmail.mc file is configured as such to allow the different milters to run depending on direction – outbound or inbound:

FEATURE(`no_default_msa', `dnl')dnl
DAEMON_OPTIONS(`Port=submission, M=Ea, Name=MSA, Family=inet6, InputMailFilters=opendkim')dnl
DAEMON_OPTIONS(`Port=smtp,Name=MTA, Family=inet6')dnl
define(`confMILTER_MACROS_HELO', confMILTER_MACROS_HELO`, {verify}')dnl
INPUT_MAIL_FILTER(`smf-spf', `S=inet:8890@127.0.0.1, T=S:30s;R:1m')dnl
INPUT_MAIL_FILTER(`opendkim', `S=inet:8891@127.0.0.1')dnl
INPUT_MAIL_FILTER(`opendmarc', `S=inet:8893@127.0.0.1')dnl

Why do the milters listen on a local TCP socket, instead of a UNIX domain socket? Simply, they don’t yet have SELinux policies in place that let them use a domain socket. Once these packages are properly reviewed and included in Fedora/EPEL, we will adjust the listening port to be a domain socket.

Of these milters, opendkim and opendmarc seem to be properly maintained still. smf-spf, for its whole ~1000 lines of code, has been largely untouched since 2005, and its maintainer seems to have
completely fallen off the Internet. All my attempts to find a valid address for him have failed. There are a variety of other SPF filters, the most popular of which is python-postfix-policyd-spf – which as the name implies is postfix-specific, and as I noted, I’m not running postfix. Call me lazy, but sendmail works well enough for me at present.

These milters are currently under review (smf-spf, libspf2, opendmarc) in Fedora and will eventually land in the EPEL repositories as well. opendkim is already in EPEL.

If you are using SPF, DKIM, and DMARC, what does your configuration look like? Please leave a comment below.

Spamfighting: Keeping spammers from using my domain

We all dislike email spam. It clogs up our inboxes, and causes good
engineers to spend way too much time creatively blocking or filtering
it out, while spammers creatively work to get around the blocks. In
my personal life, the spammers are winning. (My employer, Dell, makes
several security and spam-fighting products. I’m not using them for
my personal domains, so this series is not related to Dell products in
any way.)

I recently came across DMARC, the Domain-based Message Authentication,
Reporting & Conformance specification. One feature of DMARC is that it
allows mail receivers, after processing given piece of mail, to inform
an address at the sending domain of that mail’s disposition: passed,
quarantined, or rejected. This is the first such feedback method I’ve
come across, and it seems to be gaining traction. Furthermore,
services such as dmarcian.com have popped up to act as DMARC report
receivers, which then display your aggregate results in a clear
manner.

A DMARC-compliant outbound mail server provides several useful bits of
information. 1) The domain publishes a valid Sender Policy Framework
(SPF) record. 2) The domain signs mail using Domain Keys Identified
Mail (DKIM). These are best practices now, in place by millions of
domains. In addition, the domain publishes its DMARC policy
recommendation, what an inbound mail server should do if a message
purporting to be from the domain fails both SPF and DKIM checks. The
policies today include “none” – do nothing special, “quarantine” -
treat the message as suspect, perhaps applying additional filtering or
sending to a spam folder, and “reject” – reject the message
immediately, sending a bounce back to the sender.

A DMARC-compliant inbound mail server validates each incoming message
against two things: compliance with the Sender Policy Framework (SPF)
and checks the DKIM signature. The server then follows the policy
suggested by the sending domain (none, quarantine, or reject), and
furthermore, reports back the results of its actions daily to the
purported sending domain.

I’ve been publishing SPF records for my personal and community
organization domains for several years, in hopes this would cut down
on spammers pretending to be from my domains. I recently added DKIM
signing, the next step in the process. With these two in place,
publishing a DMARC policy is very straightforward. So I did this,
publishing a “none” policy – just send me reports. And within a few
days, I started getting reports back, which I sent to dmarcian.com for
analysis.

What did I find?

On a usual day, my personal domain, used by myself and family members,
sends maybe a hundred total emails, as reported by DMARC-reporting
inbound servers. My community org domains may send 1000-2000 emails a
day, particularly if we have announcements to everyone on our lists.
That seems about right.

In addition, spammers, mostly in Russia and other parts of Asia, are
sending upwards of 20,000-40,000+ spam messages pretending to come from my
personal domain, again as reported by DMARC-reporting inbound
servers. Hotmail’s servers kindly are sending me reports for each
failed message they process thinking they were from me – a steady
stream of ~3600/day. No other DMARC servers have sent me such forensic
data yet.

spamsources

Spam source by country for the last week

For several days, I experimented with a DMARC policy of “quarantine”,
with various small percentages from 10 to 50 percent. And sure
enough, dmarcian reports that the threat/spam mails were in fact
quarantined. It was really cool to wake up in the morning, check the
overnight results, and see the threat/spam graphs show half of the
messages being quarantined. It’s working!

However, dmarcian also reported that some of my legitimate emails,
originating from my servers and being DKIM-signed, were also getting
quarantined. What? That wasn’t what I hoped for.

It turns out that authentic messages were in fact being forwarded -
some by mailing lists, some by individuals setting up forwarding from
one inbound mail address to another. Neither of which I can do
anything about.

This isn’t a new problem – it’s the Achilles heel of SPF, which DMARC then inherits. Forwarding email through a mailing list typically makes subtle yet valid changes while keeping the From: line the same.

The Subject: line may get a [listname] prepended to it. The body may
get a “click here to unsubscribe” footer added to it. These
invalidate the original DKIM signature. The list may strip out the
original DKIM signature. And of course, it remails the message,
outbound using its own server name and IP, which causes it to then
fail SPF tests.

Sure, there are suggested solutions, like getting everyone to use
Sender Rewriting Scheme (SRS) when remailing, and fixing Mailman and
every other mailing list manager. Wake me when all the world’s email
servers have added that, I will have been dead a very very long time.

So, I switched back to policy “none”, and get the reports, aggravated
that there’s nothing I can directly do to protect the good name of my
domains. It’s hard both knowing the size of the problem, and knowing
we have no technological method of solving it today. Food for
thought.

In part 2 of this series, I will describe my system setup for using
the above techniques.

Do you use SPF? Do you use DKIM? Do you publish a DMARC policy? If so, what has your experience been? Leave comments below.

F/OSS for querying publicdata.com?

Has anyone written free/open source software to use the publicdata.com query API? I’ve got several dozen coaches we need to do background checks on, and we use publicdata.com. Turns out our folks have been doing this manually for years. Seems like something that could be easily automated, but haven’t found any software, open or otherwise, to do so.

s3tools / s3cmd needs a new maintainer

As posted to the s3tools-general mailing list, s3tools maintainer Michal Ludvig is looking for new maintainers to step up to continue the care and feeding of the s3tools / s3cmd application.  s3cmd is widely used, on both Linux and Windows, to publish and maintain content in the Amazon Web Services S3 storage system and CloudFront content distribution network.

I use s3cmd for two primary purposes:

  1. as Fedora Mirror Wrangler, I use it within Fedora Infrastructure to maintain mirrors within S3 in each region for the benefit of EC2 users running Fedora or using the EPEL repository on top of RHEL or a derivative.  Fedora has mirrors in us-east-1, us-west-1 and -2, and eu-west-1 right now, and may add the other regions over time.
  2. for my own personal web site, I offload storage of static historical pictures and movies so that they are served from economical storage and not consuming space on my primary web server.

I congratulate Michal for recognizing when he no longer has the time to commit to regular maintenance of such an important project, and to begin looking for contributors who can carry out that responsibility more effectively.  While I’ve submitted a few patches in support of the Fedora Infrastructure mirror needs, I know that I don’t have the time to take on that added responsibility right now either.

If you use s3cmd, or have contributed to s3cmd, and feel you could make the time commitment to be the next maintainer, you’ll find an active contributor base and dedicated user base to help you move the project forward.

 

SELinux on a Rackspace Cloud Server

After a long time hosting my personal web site at WestHost, I finally decided to move it to another cloud provider – a Rackspace Cloud Server.  This move gives me a chance to run Fedora 16, as I do at home everywhere, and which is more than capable of serving a few light traffic domains, personal mail and mailing lists, and email for our neighborhood youth basketball league.

One thing that surprised me though was that the default Fedora 16 image provided by Rackspace in their smallest configuration (256GMB RAM, 10GB storage) had SELinux disabled, and no selinux-policy package installed.  Being a big fan of Mark Cox’s work reporting on vulnerabilities in RHEL, and Josh Bressers work leading the Fedora Security Response Team, it just didn’t feel right running an internet-facing Fedora server without having SELinux enabled.

This was easily enough resolved by installing the selinux-policy-targeted package, editing /etc/grub.conf to remove selinux=0 from the kernel command line, enabling the configuration in /etc/selinux/config, and restarting the server.  After a few minutes of autorelabeling, all is well and good.

I’m sure SELinux can get in the way of some application deployments.  It’s easiest for Rackspace to keep it disabled, letting experienced folks like myself enable it if they want.  I would have preferred it to be enabled by default, as there’s always the option to disable it later or run in permissive mode.

Because I run a few mailing lists using mailman, across multiple domains, I of course wanted to run several separate instances of mailman, one for each domain.  Fedora has a SELinux-aware mailman package just a quick yum install away.  The problem is, the SELinux file context rules are written expecting only one instance of mailman per server.  That’s when I remembered a recent blog post by Dutch where he had patched the mailman spec and config files to build separate mailman-${sitename} RPMs, each with their own correct SELinux contexts.  Very cool, and exactly what I needed.  Well, almost – he did his work on EL6, I’m running Fedora 16, but close enough (see his blog comments for the few changes necessary on F16).  Thanks to Dutch, I’ve got a fully SELinux-secured web and mail server with separate mailman instances for each domain.

Next time you build a Rackspace Cloud Server running Fedora, take an extra couple minutes and enable SELinux.  The site you save may be your own!

FUDCon Blacksburg videos

I shot videos of several of the presentations at the Fedora User and Developer Conference yesterday.  For your viewing pleasure:

  • “State of Fedora” from the Fedora Project Leader, Jared Smith [ogg]
  • Mike McGrath, team lead for OpenShift, demoing OpenShift [ogg]
  • Jon Masters and Chris Tyler, on the ARM architecture in Fedora [ogg]. ARM is a secondary architecture today.  By Fedora 18, with your help, it needs to become a primary architecture.
  • David Nalley presented on CloudStack, which is aiming for Fedora 17 inclusion. [ogg]
  • Dan Prince and Russell Bryant giving an introduction to OpenStack [ogg]
  • Mo Morsi presenting the Aeolus cloud management project [ogg]

[Update 1/18/2012] I was able to upload all the videos to YouTube.  http://www.youtube.com/playlist?list=PL2BAA7FF83E6482C2
is a playlist with all 6.

Northwest Austin Youth Basketball registration

Northwest Austin Youth Basketball Association (NWAYBA) registration deadline is only 3 weeks away.  Register your 1st grader through High School player, and join us on the courts this Fall.  Registration forms must be postmarked by October 16, but I’d appreciate it if you’d mail them sooner.  Somehow I got roped onto the NWAYBA Board, as the Registrar.  We’re expecting 400 players again this year. I’d prefer not to deal with 300 applications in the last week.

Central Texas Wildfire Relief Food and Goods Drive

On Saturday, September 24th, the boys of Cub Scout Pack 2 will be in the Doss Teachers’ Parking lot from 8a until 12p for final collections before delivering the food and goods to those in need. Firefighers with a Fire Engine from the City of Austin Fire Department will join us at Doss from 9-11am, work permitting.

Please demonstrate the generosity and caring of our neighborhood by joining Cub Scout Pack 2 in collecting the following needed items for those impacted by the wildfires of the past several weeks:

  • Canned food items
  • Dishwashing soap
  • Diapers and wipes (all sizes)
  • Hand sanitizer
  • Eye drops
  • Bandages
  • Neosporin/triple anti-biotic cream
  • gift cards or cash which will be converted to HEB gift cards

Boys from Pack 2 have been and will be in uniform at Doss each day September 19th through the 23rd to collect your donations. So far several truckloads of items have been collected.

Google Voice: Why do I need a home phone?

For the past 3 months I’ve been using Google Voice, and I must say, I like it.  But I’m not exactly using it as intended.

I’ve had the same home phone number for 10 years.  A lot of people have that number.  Not a lot of people call it (what that says about my popularity I don’t really want to know), and we don’t make that many outgoing calls a month, but the thought of changing it everywhere is daunting.  More so for anyone with a number for even longer.  I’ve started doing so, but only opportunistically.

What to do?  I don’t want to give up my home number, and I can’t yet transfer my number to Google Voice.  And in theory, I get a discount on my phone/cable/internet by having all three, they’d charge even more for having just two.

My trick?  Time Warner offers unlimited free call forwarding.  So, my home number forwards to GV.  GV then forwards to my cell phone, email, Celeste’s cell phone, etc.  I dropped the voicemail from TW, as now GV takes care of that.  And I can drop the long distance with TW and use GV for that too.  Everything works great.

At some point, when I can transfer my home number to GV and have two numbers for the account (old home number and new GV number I’ve been giving out), and if TW’s rates change again so it’s cheaper to drop their phone service, I will.  Or they will get enough competition to realize that for a couple dozen calls a month, charging $$ for phone service won’t work and they just throw it in for free.  Here’s to hoping.