This page is to explain my experiences and tips and techniques for using (or not using) WebCatalog as announcement list server. I've broken it down into four subsections, over two web pages.

1. Sending mass mail basics
2. WebCatalog's [sendmail] tag
3. Using other products
4. Code samples

Sending Mass Mail Basics
So you have a list of email addresses, and you've seen that WebCatalog can send an email, and you're thinking to yourself, "Why don't I just use WebCatalog as an email merge engine to send out a mailing from my customer database..." Sounds good, it even sounds pretty easy. But let's back up and learn some basics about email, and bulk email.

Email consists of two parts: the envelope, and the message. All you ever see is the message. Even the headers, which you email program may show you, is part of the message. The envelope is a little extra data, that accompanies a message, that is used by one mail server to tell another mail server who the message is for. You might think the to: tag (or the cc: tag) does this, but they don't. Have you ever received an email, where it wasn't sent to you (meaning your email address isn't listed in the to: field) so you examined the email headers, and even there, your email address isn't listed? That's because the message was sent with your email address (probably along with 50K others) in the bcc: field. When the program that created the message contacts a mail server, it says hello (they really are quite friendly) and then asks the mail server if it will receive a message for a list of email addresses, the server agrees (or it can disagree, in which case nothing happens) and then the server says to start sending the message. Look at what just happened. The email program can stick any old email address in when it asks to send an email, these addresses can have something, or nothing to do with what you see in the to: field. The mail server then creates an envelope, and awaits for data for the message body. That to: field you see in the message body --- meaningless. The first mail server looks up (via DNS) to see who the next mail server should be, it contacts that next mail server, and the process repeats.

Why does this matter? Unless you have a very small (under 500) email address list, it is impractical to send an individual email to each person on the list, using WebCatalog. (We will go into this below!) You will have to use bcc's to have any efficiency, and you will encounter some problems. Whenever you send out a mass mailing, you will get a response. No not traffic to your website (although that is probably the point of your email, and it may also occur) but emails that come back to you. I'll make it simple, expect a 5-10% rate of return emails whenever you send an announcement type email campaign. (100 addresses = 10 returns:manageable, 1000 addresses = 100 returns:time consuming, 10K email addresses = 1000 returns: uhh Houston, we have a problem.)

So what are these emails that you will be getting sent back to you? There are a few different kinds of response emails:

• Invalid email addresses - some addresses are invalid because they were entered wrong (typos) were intentionally bad addresses (@a0l.com) or have just been deactivated over time. You'll typically receive one of two types of return emails, depending on the error. If the domain is invalid, you'll get a error from your email program, or your mail server saying the message could not be delivered. If the domain is valid, but the mailbox no longer exists, then a short while after sending the email, you should receive back an email containing an error message from the receiving mail server informing you that the mail could not be delivered (and maybe, if you're lucky, the error message will list which  email address that the email was sent to, was not valid). In any case, these email addresses should be removed from the list. Using bcc's to send email will eventually drive you nuts, as the bounces will come back, and you will not be able to cull enough info to figure out what address is bouncing.
• Vacation notices - eventually you'll notice that some people never return, and should be removed from the list.
• "Your message has been received / opened / read / woven into a nice asbestos car muffler doily." Useless, this stuff is response spam.
• Customer service email. In the huge piles of returns you get, there will always be a few people who reply to the message and this gets in with all of the bounces.
• Sometimes you'll even get mail about this and that unrelated topic - I'm not making this up, someone wrote back and complained about their computer speaker emitting a tone ever since they got a piece of marketing email!

and of course, my personal favorite... (drum roll please...)

• "REMOVE ME NOW YOU FRIGGIN $%@&#*#! This is my 237th request, since I started sending you requests about 30 seconds ago to REMOVE ME from your list, and I have yet to receive a certified letter signed by the president of your company, and your local congressman assuring me that I will never ever receive any additional emails from you. Oh, and you should also send me some flowers and a box of chocolates for the LARGE inconvenience of receiving the last email you sent me. It took almost .5 microseconds to delete it as I was scanning through my inbox......."
  As much as you can shout, "go to our web page to change your subscription (change address/unsubscribe)" about .5% of you list will reply to the sent from email address with "unsubscribe/remove/various profanity/claims of unsolicited spam." Multiply .5% times the size of your list. If you have 100 names, it won't be much, if you have 100000 names, all bets are off. These messages will drive you nuts, as the email address they send you this complaint from will not be on your list. (I'll be more then happy to say "I told you so" on this point in the future if need be.)

One problem to be aware of with sending mail via bcc: is that mail will get forwarded (this is a server side bcc) and the new recipient will have no idea why they are receiving it. An example, TimJ@BigCompany.com was laid off, so they set up a little forward on their server to send his mail to his replacement (along with the mail of 25 other recently laid off people) to MarshaB@BigCompany.com, You will get a nasty email from Marsha, who is wondering why you are sending her spam. You will never find her name in your list, because technically you aren't sending the email to her.

WebCatalog's [sendmail] tag
First and foremost, I AM NOT using WebCatalog to send email for the simple reason that it CAN'T. I am using WebCatalog to write text files to disk which another program (Emailer) comes along and sends as email. This document refers to WebCatalog on a MacOS (not MacOS X) server. For Windows NT and the various flavors of Unix, I appreciate your input on your experiences!

My first attempts to use WebCatalog as a announcement mailing list server were back around May of 1998. For you Mac users, this was back in the days of normal HFS hard drives (not HFS+) which had very hard times with directories of more then about 1000 to 1500 files. This is important to note because of how Emailer works, and how computers OSes deal with having a large number of files in one directory.

WebCatalog sends email by writing a file to the hard drive. One file per email to be sent. On the Mac, Smith Micro includes a little program called Emailer. This program scans the folder where WebCatalog writes out the email files, and it reads each file individually, and attempts to contact a mail server, and deliver the mail to the mail server (typically your in-house or ISP's mail server) Then Emailer moves the text file to a "completed" folder. Simple right. So if you want to send a few emails, you only need to write a few files, and everything happens automatically, and it's done. Lather, Rinse, Repeat. And then you find yourself with 500 names on your list, and then a 1000... Emailer is single threaded, so it has to wait to send each piece of email to the server. This shouldn't put too much of a burden on your email server, Emailer isn't that fast. But if you have a few hundred emails, it's going to take a while to get them through the process. If you have any order confirmation email, or Thank You emails, they get put at the back of the queue. So what if they are delayed a few hours?

In all fairness, I'm used to lists with 10K or more email addresses (the biggest I used to play with had 300K+.) If your list has 200 addresses, then most of this won't matter, but read on anyway, hopefully your list will grow, you should be aware of some limitations...

Eventually, you will hit upon a performance limitation of putting one email address on each message and sending them individually. Here's the problem: when you send an email, WebCatalog has to write a file to the disk to send the email, 10K emails = 10K files in one directory. BAD IDEA. At that point, You'll think, why not use the bcc: field. Been there, done that. Actually discovered a bug in Emailer using the bcc email tag. WebCatalog (or more particularly the Emailer program (on the Mac anyway) has a small and annoying flaw. To wit: you will get double Bccs if you don't put in a cc: The cc: can be blank, i.e. to=me@here.com&cc=&bcc=(your list) I don't know if this problem (and solution) still exist in the latest versions of WebCatalog's Email, but if you're having problems, give it a try.

So you switch over to bcc'ed email. If you are using the standard WebCatalog Emailer, then you will probably have 50 or 100 Bccs per message, if you have 10000 email addresses, you will have 200 files written (at 50 addresses per). These then litter up the email completed folder, which starts to grow because no one is watching it. This eventually causes directory damage to the hard drive (When I started, I didn't know this, and I ended up trying to clean 43000 sent emails out of this directory on one of my servers.)

Then these bcc emails hit the mail server. It isn't pretty. Each message that Emailer sends to the server breaks open into 50 individual emails, Now most email servers are smart, and try to group all of the messages going to one domain for a bigger batch send, but typically when you send your mailing list, you don't have a way of sorting by domain, to help the mail server. The mail server also relies on the DNS server, and I hope your DNS name server is up to speed too, cause all those name lookups are going to place quite a load on it.

So you start sending emails via the bcc method, and after a while, you eventually get tired of the remove complaints, (or decide you want to send individual customized messages) to each individual directly. The individual messages have true power, because you can put in traceable links. But, we're right back where we started concerning the limitations of sending individual emails. Oh, have you ever seen you server come screeching to a halt as it wrote 15000 files to the hard drive? Not a pretty sight. (And then the customer complaint phone starts ringing.)

So what's the answer? For a small list, WebCatalog, writing out files like it does is probably ok, Other then that, you might want to take a look at some of the third party offering below.

Here are some other random thoughts I have about this:

Is your webserver a webserver, or a bulk mail server, do you want people to leave your site because it's annoyingly slow because it's busy sending out mail?

• a &temp=T modifier to delete mail that was successfully sent. I don't want this universally, because I keep the thank you and fulfillment email for redundancy. Bulk promotional mail on the other hand just fills up the EmailCompleted folder. (I might delete the thank you mail too, come to think of it)
• Have Emailer move the unsuccessful email to a different folder...
• An ability to give Emailer 2 files, a message file, and a list of email addresses, and have it do the email merge.

Using WebCatalog to send one-to-many emails isn't going to go away. Other commerce software (including Microsoft commerce server) has in the admin pages of a shoppable website the ability to administrate an email promotional list. I think it would be a short coming of WebCatalog not to have something similar.

Third Party Options
I've heard that Emailer can be replaced by SIMS on the Mac.(Ken Grome is the Mac Daddy for this, I'm lead to believe) I understand that you still write individual files, but SIMS handles delivery, then deletes the messages. "Have you ever seen you server come to a screeching halt as it wrote 15000 files to the hard drive?"

If you're on a Mac, or have access to one, I can think of no better program to send bulk email then this one: eMerge by Galleon Software. I do NOT work for these people, and Yes, I did buy a copy... anyway, it's pretty cool, as you can write an email using variables and send a distinct copy to individual subscribers. (Including - finally - their email address in the body of the message) it also has it's own built in DNS resolver, and opens a connection directly to the recipient's mail server. (Read: no load on your DNS server or mail server). You can't find a program for sending bulk email that has a better Mac interface. This does, however, depend on you writing your email address list to a big file, shortly before you want to send your message, and then on another computer (not the server) running eMerge to merge your letter/offer and the email address list and send it out. It is a nice companion product for WebCatalog, and it is a measly $99! Buy it soon before they realize the same functionality (with a much worse user interface) on the PC will set you back about $399.

For eMerge, or coding your own solution, you will probably want help managing the returned email. There aren't many programs better then Smartbounce. It's a little thorny to configure, you have to be somewhat handy editing a text based preferences file, but once it's set up, you can use it weekly with not too much thought or effort. They used to have a free version, but a recent check of their site now shows they have gone the way of commercial only software, and they must be ashamed of their pricing, because they don't list it on the site. Sorry...

For faster delivery speeds, and built in Bounce Management, you can try a rather expensive solution called Lyris. It's windows based, and it is a complete mailing list server, but it was designed for people who need to send a lot of email, with a minimum amount of hassle.

Sample WebCatalog Source code
Either you have a small mailing list, and are content to live within the limitations of WebCatalog, or "Brian Burton? Brian B. Burton? You're a jerk Burton, a real kneebiter" you have a medium size or big list, and think I'm lying to you, so you want to prove me wrong.

In either case, the code you're looking for is here...