Last quarter, newsletter giant The Hustle ran a test that got 43,876 additional clicks on email.
Not a total of 44,000. 44,000 apart.
If you’re anything like me, you’d do something awful for that kind of engagement.
So I knocked on the managing editor’s door and demanded to know his secret.
Fortunately, the process is simple and repeatable – nothing too scary. So that’s the crowbar, because I’m going to show you a process that you can start today.
How-To For The Same What
Ben Berkley, The Hustle’s managing editor, is constantly grinning. Which is fitting, because behind that grin is a very simple method – with a few twists to come.
Here’s how it works:
Every weekday morning, Ben cooks up two separate subject lines for each newsletter. At 5:30 am ET, each subject line was delivered to only 30% of The Hustle’s audience.
After an hour, the variant with the most engagement is automatically sent to the remaining 40% of recipients.
“Is that it?” you think “It’s just a stinky A/B test?!”
Not so fast. I promise you twists:
- These variations are sent to four predefined customer segments.
- The winner is based on click through ratio (CTR) – no open rates.
- Ben’s secret sauce: The WTF factor.
Let’s dig in.
Twist 1: Segmenting Readers
Hustle splits the two original test emails between them four part.
To help me understand this, Ben referred me to Kaylee Jenzen, Hustle’s resident MarTech mentor.
(“If the Nobel Foundation ever adds a prize to the newsletter, I’ll nominate Kaylee in no time,” he said.)
“Hustle uses HubSpot’s workflow to segment contacts with values from 1 to 4,” Jenzen explains. “Email 1 is targeted at segments 1 and 2, while Email 2 is directed at segments 3 and 4.”
In other words: One email, two variations, four sent. The reason?
“By spreading out email submissions over time, The Hustle avoids spam filters and ensures more consistent delivery rates,” says Kaylee.
Takeaway: High sending volume is a signal that email service providers use to identify spammers. By reducing your recipient list and increasing your send times, you make your email more secure in the eyes of your email server.
Twist 2: Targeting CTR, Not Open Rate
If you’ve tried A/B testing subject lines before, I’ll bet you based the test on the open rate, right?
That’s what almost every tutorial will tell you, and it’s not a bad choice. But it’s not only choice.
While open rate measures the number of readers who access an email, it cannot separate readers who delete immediately. Click-through rate, on the other hand, measures how many readers actually engage with the email.
“By emphasizing CTR, The Hustle aims to not only engage readers, but also encourage them to take action and interact more with the content,” Jenzen said. “This approach represents a strategic effort to push meaningful engagement beyond mere disclosure.”
See, The Hustle’s open rate is already higher than the industry average, so changing the subject line only made a 1% difference in that metric. (#HumbleBrag?) So the team made a strategic choice to focus their time and efforts on developing deeper engagement.
Takeaway: Open rates can be the metric you want to focus on. But think about your ultimate goal. Optimize to get it singnot what YouTube tutorials tell you.
Round 3: The WTF Factor
“When people search their inbox, they often just want to delete all the new stuff,” says Ben. “And they got a rhythm.”
The rhythm is the sound of your email being deleted.
“So the goal is to gently take people out of rhythm,” he said.
That’s where Ben’s secret sauce comes in.
“To do that, you need a ‘WTF?’ factors sprinkled in.”
As the former executive editor of The Onion, Ben is no stranger to the WTF factor. But that doesn’t mean he’s trying to recreate the Onion title for The Hustle.
“I’m never looking for shock value or trying too hard to throw people off,” he warns. “But you eventually want them to see the unexpected and give them a quick break. You just want them to glitch for a split second.
Some of my favorite examples:
- 🎨 Believe in yourself, but not like that
- 🕹 Buried under cement in a landfill
- 🔓 Break out of the toothpaste jail
Takeaway: Your subject line should be compelling enough to shake you off autopilot and spare a few minutes for your email.
But what does that really mean?
How to Write a Hustle-Like Subject Line
Ben shared with me what he calls the “Subject Line Manifesto.” These are the real guiding principles for The Hustle’s editorial voice.
1. Be unexpected.
You are not likely to jostle people out of rhythm with a subject line that emphasizes something they already know.
“Having worked in comedy, I understand the power of subverting expectations,” says Ben. “This is the essence of laughing and creating something unforgettable.”
“Besides all the unforgivable crimes, there is only one unforgivable crime in The Hustle,” he added. “That’s boring.”
That doesn’t mean you have to try to be funny. In fact, it may not be a good fit for your brand. You can change expectations with new data, surprising ideas, or thought-provoking questions.
What part of your email adds something new to the conversation? That’s your subject line.
And speaking of…
2. A strong subject line comes from a strong story.
“The subject line should match the overall vibe of the newsletter, as well as the specific content of each day,” Ben explained. “So really, writing a strong subject line starts with writing a strong story.”
So if your email not add something new to the conversation, maybe pause the subject line for now and revisit your content strategy.
3. Don’t laugh cheaply.
If you decide to use humor, don’t use the same words as 10 of your competitors. Chances are, examples just popped into your head. Skip that.
“The low-hanging fruit in comedy is low-hanging fruit for a reason; it’s there because that’s what people want,” Ben added. “If you give it to him, you can have a light laugh, or a laugh or two, but you won’t laugh because he’s not surprised. Valuable content comes from jolts.
And while we’re on the subject…
4. Don’t settle for clickbait.
“It’s so easy for your subject line to sound like clickbait, and that’s the last feeling your audience wants,” Ben says. “In this hypercompetitive attention economy, if you break that relationship, you can’t get it back.”
5. Keep it brief.
Hustle sees performance improvements from subject lines that are only 11-15 characters long.
The shortest subject line enjoys almost the same level of engagement 75% higher rather than the longest email.
“When I approach a subject line, I always remind myself: The words are there to heal the feeling, not to transcend it,” says Ben. “You have to give them enough context to get them interested, but not many people feel like they have the whole story.”
Now that you’re a subject line expert, let’s talk process.
How to Take a Test at Home
Sending multiple emails a day seems like a lot of work, and it is shall so if you do it manually. Fortunately, most email marketing tools have some sort of automatic A/B testing feature.
The Hustle happens using Marketing Hub, so we’ll show you how to do it in HubSpot, but the process should be pretty much the same no matter what tool you use.
1. Create an email as usual.
In HubSpot, like most tools, this will automatically be the A version of the A/B test. (So name it, and give it your first subject-line variation.)
2. Click on “Create A / B test” in the upper left corner.
3. Enter a name for version B, then click “OK.”
Don’t overthink this. Naming conventions are one safe place to show your boring side.
Even Ben apologized in the picture below.
4. Edit the subject line for version B.
Remember that the best practice for A/B testing is to only change one element per test. Otherwise, you risk skewing the results.
5. In the version drop-down menu, click “Manage tests.”
6. Adjust the A/B distribution for the desired mix.
Hustle uses 30% of the recipient for each variation, with 40% reserved for the winner. But with more than two million subscribers (#HumbleBrag confirmed), it can spare some readers to get more accurate results.
If you have a smaller volume of posts, you might want to stick with 20-25% for each test version. While your test will be based on a smaller sample size, proven winners will appear to more recipients.
7. Set the winning metric to “Click through Rate.”
To emulate The Hustle method, you’ll want to choose CTR. That said, consider the goals of your campaign.
The tool also allows you to choose “open rate” or “click rate.”
8. Set “Test duration” to 1 hour.
You will see the danger of setting a duration of “at least 4 hours” in order to achieve conclusive results.
That’s good advice for most email marketing, but it can be problematic for newsletters. HubSpot’s research shows that the best time to post is between 9 a.m. and noon Eastern time.
You can see why a four-hour test can be complicated. But what you will lose in statistical significance, you will make up in eyeballs.
9. Click “Save changes.”
10. View email settings and recipients as usual, then send.
Make Your Hustle
To cheer up our legal department and keep you from freaking out me door: I can’t promise you 43,876 additional clicks.
But by following a data-driven approach, you’re almost guaranteed to see improvements. Pair that with Ben’s advice, and you’ll be bugging your readers on a regular basis.
At least you won’t be bored.
