# data[].attributes.body
There are many tactics you can use to gently sway your organization to become more experimentation driven. Here are 5 ideas…

# data[].attributes.attachment_link.readable_text
Conversion optimization is hard, but the heaviest lifting is often organizational - changing an organization from a gut-based approach to one driven by data and experimentation.

Picture, perhaps, a Director of Marketing at a large, stable company with a solid brand. She's on the cutting edge of digital marketing, reads articles like this and wants to implement a testing culture within her team and organization. Problem is, HiPPOs at her company are staunchly opposed due to a variety of entrenched ideas and political woes.

Or perhaps you're heading growth at a fast-growing startup and want to build a culture of testing and continuous data-driven growth. How can you build a culture of experimentation, when some executives opinions are so strong and their beliefs so entrenched?

How Small Tweaks Can Bring Big Change

Richard Thaler and and Cass Sunstein wrote in Nudge about a concept called " choice architecture" - swaying decisions towards the best option through a variety of small tweaks called "nudges." Things as seemingly insignificant as making organ donorship the default (as opposed to needing to opt-in) create huge changes in results.

Similarly, in Switch, Chip and Dan Heath wrote that, "Big-picture, hands-off leadership isn't likely to work in a change situation, because the hardest part of change - the paralyzing part - is precisely in the details." They detail many such examples, like Microsoft requiring developers to program on the same machines customers use to facilitate better usability for the end user, where small changes create big results.

There are many tactics you can use to gently sway your organization to become more experimentation driven. Here are 5 ideas...

Note: It's really difficult to describe in precise terms what a 'culture of experimentation' is, but I'll try. For this article, it will mean a culture that is predicated on a belief that there aren't any sacred cows in terms of the way things are done or designed. An opinion, no matter how highly paid, is still an opinion - instead, an organization makes data-informed decision. In a culture of experimentation, failure isn't punished, insights and learning are rewarded, and team members are encouraged to innovate and discover new ways of growth.

1. Script the Critical Moves

To begin, it's important to clearly define what a 'culture of experimentation' looks like. Change can only occur with the participation of individuals and their behavior, but that's a hard place to start because that's where friction occurs. In Switch, Chip and Dan Heath wrote:

"To spark movement in a new direction, you need to provide crystal-clear guidance. That's why scripting is important - you've got to think about the specific behavior that you'd want to see in a tough moment, whether the tough moment takes place in a Brazilian railroad system or late at night in your own snack-packed pantry."

An idea for optimization: make a list of 4-8 rules to guide your optimization process. For example, Andrew Anderson, Head of Optimization at Malwarebytes, has the following guidelines for a discipline-based testing program:

All Test Ideas are Fungible

More Tests Does Not Equal More Money

It Is Always About Efficiency

Discovery is Part of Efficiency

Type 1 Errors are the Worst Possible Outcome

Don't Target just for the Sake of It

The Least Efficient Part of Optimization is the People (Yourself Included)

He also mentions that you should proactively generate roadmaps and focus areas. While this isn't a small amount of work, investing in a winning process ensures a successful testing program (and there's no better way to get buy-in than to produce ROI).

Specific, transparent, and publicly displayed choice architecture ensures clarity, and as Chip and Dan Heath wrote, "Clarity dissolves resistance."

Clarity can be achieved in many different ways. Wherever you can script the critical moves and design the path with clarity, you should. One example that comes to mind is metrics. What's the one metric of success that will define your tests? Defining it ahead of time will help eliminate political upheaval in the case of an ambiguous test.

Twitter does this to prevent HARKing. As they wrote:

"One way we guide experimenters away from cherry-picking is by requiring them to explicitly specify the metrics they expect to move during the set-up phase. Experimenters can track as many metrics as they like, but only a few can be explicitly marked in this way. The tool then displays those metrics prominently in the result page. An experimenter is free to explore all the other collected data and make new hypotheses, but the initial claim is set and can be easily examined."

Matt Gershoff, CEO of Conductrics, also advises clarity upfront when it comes to expectations.

Matt Gershoff:
"Before doing anything, the team should first articulate what the expected gain in terms of value (revenue, gross profit) might be in the best case, and what the loss might be in the worse. This way you can filter out ideas that even in the best case really won't have much of an impact to the business as well as ideas that have a good chance of failing badly, which could demoralize the organization."

2. Commit To a Testing Cadence

When beginning a testing program, a lack of results can be frustrating, but Sean Ellis, founder of GrowthHackers, has found success in planning and maintaining a specific test cadence. Here's he explained it:

Sean Ellis:
"For me the main thing that creates a culture of experimentation is committing to a testing cadence and sharing results. In the beginning you may need to be patient to generate results. But I've never seen a company run 10+ highly considered tests and not achieve a meaningful improvement. Wins drive buy in and buy in accelerates testing momentum.

So for me, the most important first step is committing to a weekly experiment release schedule. Stick with it for at least a month. Sharing result will drive more company wide participation. Over time you'll find that the whole process is pretty addictive. But it requires a commitment and perseverance in the beginning."

This is important in the beginning, but also for maintain momentum as your program matures. As Peep wrote, your impact is determined by three factors:

1. The number of tests you run

2. The number of winning tests

3. And by how much each test wins

While tempo alone doesn't affect the latter two (this article tells you how to have more winning tests, and larger wins), it assures you'll enough tests to make a good impact. And it ensures some wins - important for buy-in, especially early on.

3. Make it a Game; Trigger Healthy Competition

Using Projects by Growth Hackers, I noticed they had a feature called the "leaderboard." This lets you track which members of your team are submitting the most test ideas, as well as which team members have the most wins.

While not everyone agrees that putting a focus on test ideas is efficient, competition has been known to increase productivity in certain settings. In Growth Hackers' own experience, they found that it increased the volume of test ideas submitted per week:

"After the ideation started to slow down, we decided to add a leaderboard to celebrate the people who were generating the most ideas. The leaderboard re-energized the ideation process, so that we are now adding about 2 - 3X more ideas per week than we can test. Therefore, we are continuing to grow our backlog of hundreds of ideas."

It's not just about competition, either. Social influence is a trigger here as well. As Thaler and Sunstein wrote in Nudge, "The general lesson is clear. If choice architects want to shift behavior and to do so with a nudge, they might simply inform people about what other people are doing."

Note that volume of ideas isn't a metric that will be important to everyone. Volume of growth ideas for a fast growing startup? Totally awesome. For our internal growth at CXL, the more ideas the better.

But for landing page optimization on a high traffic website? Sure, the more tests the better, but only if they are efficient and don't incur opportunity costs (ie you're testing dumb things and could be testing things of higher impact). Projects by Growth Hackers also includes prioritization of ideas, so don't take this trigger as a green light to try all 100 ideas on your "100 A/B test ideas to try right now" article.

4. Celebrate Failure Publicly and Often

In the early stages of developing a testing culture, your program is much more susceptible to disintegration with the arrival of any mishaps. That could be any number of things...failure to move the needle, failure to bring key stakeholders on board, failure to generate excitement, failure to renew your testing tool, and failure in its broadest sense.

To combat that, failure, or at least the insight gained from it, needs to be something that is celebrated.

Joanna Lord, Marketing Executive and Tech Advisor, talked at CTA Conference 2015 about a variety of ways she implemented this in her previous role at Porch. One way of celebrating failure was by drawing attention to it...

Joanna Lord:
"At Porch, we do a lot of really bizarre things. One thing is we have this pink, pony-dog-creature-fuzzy-animal thing - not alive, that would be cool - named Mr. Sparkles.

And every time you break the site, whoever breaks the site the worst gets Mr Sparkles for a week. You put him on your desk and it's like this badge of honor that you like did something so bold that you literally messed up the site badly.

And you know what I love? You see my CEO walk around the room and he's high-fiving the Mr. Sparkles owner. And people are like, "What did you do? What did you do to get Mr. Sparkles?

But the reality is we've made it a positive thing. We've made it a badge of honor. You are living out the Porch-y way in being bold. What can you do in your culture to make it fun and acceptable? And almost, you know, become famous for it."

While constant test failure isn't ideal (it's probably a sign you're not doing adequate conversion research), new members shouldn't feel fear at a fast growing company. Innovation can be vanquished pretty quickly in a state of underlying fear.

5. Embed Triggers Into Your Current Internal Communications

In her CTA Conference talk, Joanna mentioned another tactic they used at Porch to trigger a testing culture. Two tactics, actually, both related, and both embedded into their current internal communications.

First tactic is that they sent out weekly test roundups. These included first and foremost, the insights gained from tests, both from winners and losers. All of the other information fell below that, including whether the test one or lost. Not only does this keep communication transparent, but it spurs social influence in the organization by allowing the organization to see the scope of testing and the insights gained.

Leading with insight also changes the conversation, leading to more action instead of conceptual discussion.

The other tactic Joanna mentioned was a weekly meeting where they discuss tests:

Joanna Lord:
"We have a meeting every Friday - we call it Around-The-Porch - where we call out the tests, both winners and losers, and there's clapping and there's celebration.

So-and-so ran a test, that's why the site was down X number of hours. It was really bad - good try on the test though! Here's what we learned..and the site's back up now...And you know, it can be uncomfortable and awkward but it's so good.

And when you're bringing in new people as fast as Porch is or some of your companies, their first week in, they see people failing, and getting clapped at...? Like, this is gonna be great! I think you have to change the shift and really just shake things up."

This is a great way to celebrate testing, winners and losers, and it's a great way to trigger organizational support. It makes experimentation salient because it's embedded into existing meeting structures.

Even better, if you use a workflow management tool like Projects by Growth Hackers, you can integrate that with Slack and see all of the updates in a channel. You're on Slack all day anyway, and seeing people add ideas to the backlog, or seeing test results ready to be analyzed, triggers other team members to think about experimentation.

Sean Ellis, who spoke above about implementing the leaderboard within Projects, says that was just one of many triggers he found to be effective:

Sean Ellis:
"As for the leaderboard, it's just one of many triggers that keep people accountable to always be on the lookout for more ways to improve results. For example I started my day yesterday entering three ideas for tests. I got a thrill entering these ideas and it seems to have triggered a handful of additional ideas from other team members (the whole team was notified of my ideas over Slack)."

6. Prime Your Team Each Week With a 5 Minute Plan

Joanna mentioned both weekly digests and weekly meetings used to call attention to tests. Here's another idea for a weekly ritual, one that will both analyze the progress of your testing program and will encourage its prosperity...

Usually, when researchers send out surveys, they're simply looking for insights. But in certain cases, social scientists have discovered an odd phenomenon: when they measure people's intentions, it affects how people subsequently act. This is known as the " mere-measurement effect. "

Thaler and Sunstein consider this a nudge in the most accurate sense. For example, they noted that "if people are asked whether they intend to eat certain foods, to diet, or to exercise, their answers to the questions will affect their behavior." They also offered fascinating examples of this around voting, flossing, and fatty foods:

"It turns out that if you ask people, the day before the election, whether they intend to vote, you can increase the probability of their voting by as much as 25%...If people are asked how often they expect to floss their teeth in the next week, they floss more. If people are asked whether they intend to consume fatty foods in the next week, they consume less in the way of fatty foods."

So why not implement a 15five style survey into your weekly digest? You could ask questions like these (feel free to come up with your own):

I feel that I'm contributing to our testing culture in a meaningful way (scored 1-10)

I feel like our team successfully makes data-driven decisions (scored 1-10)

What is blocking you, if anything, this week?

Do you have any ideas for improving our growth process?

What's something you and your team did well this week?

Note: you don't have to ask those exact questions, I clearly just riffed quickly on them. But you get the point. Do a survey like this, focusing on the right things of course, and you can collect great management data as well as enact the mere measurement effect, leading your team to be more data-driven and enthusiastic about testing.

Andrew Anderson, mentioned that talking about the right things - focus areas, scope of a test - reiterates their value and makes it harder to fall into traps:

Andrew Anderson:
"This is why talking about only focus areas and the scope of a test is so important. If you make it clear the discipline is what matters and not the specifics of one experience, or even worse the specific story or "hypothesis" that someone has, the harder you make it for others to fall into those traps."

Conclusion

Organizational culture is a large and nuanced topic. This article will not, by itself, implement an efficient testing culture into your organization. That requires a variety of factors, not least the simple matter of having a team or a few allies that believe in the value of testing.

However, as has been shown in many arenas, small nudges can lead to big results, and some successful marketers have implemented some clever ideas to this effect. In summary, here are the nudges this article mentioned:

1. Script the Critical Moves (i.e. clearly and publicly define the path and goals)

2. Commit To a Testing Cadence

3. Make it a Game; Trigger Healthy Competition

4. Celebrate Failure Publicly and Often

5. Embed Triggers Into Your Current Internal Communications

6. Prime Your Team Each Week With a 5 Minute Survey/Plan

All of these won't work for all organizations, but some might work for you. Also, you can always get creative - take some of the underlying principles of nudges (defaults, social influence, transparency, etc), and come up with your ideas.

Actually, if you have any creative ideas already in action, I'd love to hear them in the comments below!

Related Posts

I recently had the pleasure of speaking with Nir Eyal, Stanford Business School lecturer, start...

Discover insider secrets on how to successfully build a high-performing growth team. How can you...

Creating a product and writing good sales copy are not enough to convert visitors into...

As a marketer and optimizer it's only natural to want to speed up your testing...

# data[].attributes.attachment_link.title
6 Clever Nudges To Build a Culture of Experimentation

# data[].attributes.attachment_link.description
Conversion optimization is hard, but the heaviest lifting is often organizational - changing an organization from a gut-based approach to one driven by data and experimentation. Picture, perhaps, a Director of Marketing at a large, stable company with a solid brand.

# data[].attributes.body
“At Porch, we do a lot of really bizarre things. One thing is we have this pink, pony-dog-creature-fuzzy-animal thing – not alive, that would be cool – named Mr. Sparkles.

And every time you break the site, whoever breaks the site the worst gets Mr Sparkles for a week. You put him on your desk and it’s like this badge of honor that you like did something so bold that you literally messed up the site badly.

And you know what I love? You see my CEO walk around the room and he’s high-fiving the Mr. Sparkles owner. And people are like, “What did you do? What did you do to get Mr. Sparkles?

But the reality is we’ve made it a positive thing. We’ve made it a badge of honor. You are living out the Porch-y way in being bold. What can you do in your culture to make it fun and acceptable? And almost, you know, become famous for it.”

# data[].attributes.headline
3M was offering employees time off to explore their own projects — netting 3M’s most famous products to date.

# data[].attributes.subject
3M was offering employees time off to explore their own projects — netting 3M’s most famous products to date.

# data[].attributes.attachment_link.readable_text
Everyone has loyalties to their favorite brands, but there's a good chance your favorite products result from two separate brands working together.

One of my own beloved childhood memories was a product of co-branding: Betty Crocker partnered with Hershey's to include chocolate syrup in its signature brownie recipe.

There's something brilliant about that co-branded product: It's a fun way to marry two classic brands into one delicious experience for fans of baking and chocolate alike. In fact, these brands still create new co-branded products to this day.

Co-branding can be an effective way to build business, boost awareness, and break into new markets, and for a partnership to truly work, it has to be a win-win for all players in the game. Both audiences need to find value - like chocolate-loving fans of Betty Crocker and Hershey's.

There are a ton of great examples of co-branding partnerships out there. To show you what makes them so successful, we've curated a list of examples of great co-branding partnerships to inspire you.

Looking to ramp up your co-branding program? Download HubSpot's Co-Marketing Kit + Templates to learn more about co-branding best practices and build out your process with universal, customizable templates.

Why is co-branding important?

One of the most important ways that people can bring value is by coming together. And that power grows when it happens at brand-scale.

Expand Your Brand's Reach

Co-branding multiplies your brand's audience with the audience of another respected brand. So, you add reach with your partner's customers and followers, and they add reach with your fans.

And, a co-branding partnership is more than the sum of its parts. Your collaboration can help both partners expand brand reach into new locations, demographics, and more. Plus it can reinforce your brand positioning.

Cost and Resource-Sharing

Marketing and branding efforts can be expensive, especially at scale. Co-branding gives both teams the resources of their partner - adding more staff, budget, and knowledge to each campaign. Co-branding also offers great learning opportunities for your team.

Improve Brand Trust and Customer Loyalty

If you have a new brand, you can boost your reputation with an established brand. This helps build trust with customers because it shows them you respect consistency. It combines their loyalty to another brand with the value they believe your brand can offer.

If you're running an established brand, partnering with a fresh startup brand shows your investment in innovation. This could translate to new and exciting products or a hot take on culture or messaging.

Boost Sales and Revenue

An expanded audience, more brand trust, and loyalty from return customers logically translate to sales. Partnerships can also offer extra revenue from new products and services that your brand might have a hard time tapping into alone.

1. GoPro & Red Bull

Co-Branding Campaign: Stratos

GoPro doesn't just sell portable cameras, and Red Bull doesn't just sell energy drinks. Instead, both have established themselves as lifestyle brands - in particular, a lifestyle that's action-packed, adventurous, fearless, and usually pretty extreme. These shared values make them a perfect pairing for co-branding campaigns, especially those surrounding action sports.

To make the partnership work, GoPro equips athletes and adventurers from around the world with the tools and funding to capture things like races, stunts, and action sport events on video - from the athlete's perspective. At the same time, Red Bull uses its experience and reputation to run and sponsor these events.

"GoPro camera technology is allowing us to complement the programming by delivering new athlete perspectives that have never been seen before," said Sean Eggert, Red Bull's former VP of sports marketing. The collaboration continues to use exclusive GoPro content to enhance both companies' growth.

Why the Stratos Co-Branding Campaign Works

While GoPro and Red Bull have collaborated on many events and projects together, perhaps the biggest collaboration stunt they've done was "Stratos," in which Felix Baumgartner jumped from a space pod more than 24 miles above Earth's surface with a GoPro strapped to his person. Not only did Baumgartner set three world records that day, but he also embodied the value of reimagining human potential that defines both GoPro and Red Bull.

2. Rachel Comey, Victor Glemaud, Sandy Liang, Nili Lotan & Target

Co-branding Campaign: High-End Fashion

Anyone who's designer-conscious knows high fashion and Target aren't exactly the same caliber regarding quality. Dresses by designers like Sandy Liang go for around $600, whereas dresses sold by Target go for more like $35 a pair. See what I mean?

But that discrepancy in pricing is exactly why these designers and Target brand decided to partner with one another. To support its brand positioning as trendy and fashionable, Target has paired with high-end fashion designers Rachel Comey, Victor Glemaud, Sandy Liang, and Nili Lotan to offer exclusive branded items for a limited time.
Image Source
Why Target Co-Branding Campaign Works

In this exchange, the high-end designers can expose their name to a new generation of potential consumers, who will increasingly aspire to own more pieces from those high-end collections while furthering the idea that Target can be a store to find fashion-forward clothing.

3. Joybird & Sherwin-Williams

Co-branding Campaign: Being Boldly Original

The Joybird home lifestyle brand and paint company Sherwin-Williams did a co-branding campaign when they partnered together to curate an eclectic collection of home furnishings and color palettes.

Together, the two brands created an exclusive product line of paints and room design inspiration that helped customers easily select paint colors to complement their furniture choices.
Image Source
Why the Sherwin-Williams Co-branding Campaign Works

These two businesses saw the opportunity to run a campaign that exposed their product and copywriting to brand new audiences. This intersection of target demographics could have brought in new business for each other and could have paved the way for future co-branding collaborations if proven mutually beneficial.

4. Balenciaga & Crocs

Co-branding Campaign: Hard Crocs

Balenciaga and Crocs have been making headlines with their fun fusion of aesthetics since 2018. But the most recent pairing got a boost from Eliot Page on the red carpet in 2022.

These brands clearly align in their dedication to innovation and individuality. As Balenciaga Creative Director Demna states in GQ "I am not interested in anything average, including the average consumer...if someone is personally offended by Crocs, there might be a more serious problem within that person than the design of a shoe."

Why the Balenciaga Co-branding Campaign Works
Image Source
This high fashion brand uses Crocs' accessible shoe brand to expand and reinforce its experimental ideas. Then, it places those experiments in public spaces to create excitement and engagement with the audiences for both brands.

5. Kanye & Adidas

Co-branding Campaign: Yeezy

Kanye West partnered with Adidas in 2015 to develop a high-end footwear line called Yeezy. The combination of Kanye's personal brand and Adidas' growing streetwear segment made for robust company earnings and brand growth.

But the brand cut ties with Yeezy in October 2022 due to harmful remarks and behavior, leading to a $655 million loss in sales at the end of that year. The brand began selling Yeezy merchandise again in May 2023, donating some of the proceeds to social justice organizations.

While initially an excellent co-branding partnership for both partners, some might also see this as a brand partnership fail.

6. BMW & Louis Vuitton

Co-branding Campaign: The Art of Travel
Image Source
Car manufacturer BMW and designer Louis Vuitton may not be the most obvious pairings. But if you think about it, they have a few important things in common. If you focus on Louis Vuitton's signature luggage lines, they're both in the business of travel. They both value luxury. And finally, they're both well-known, traditional brands that are known for high-quality craftsmanship.

These shared values are exactly why this co-branding campaign makes so much sense. In their partnership, BMW created a sports car model called the BMW i8, while Louis Vuitton designed an exclusive, four-piece set of suitcases and bags that fit perfectly into the car's rear parcel shelf.

Although the four-piece luggage set goes for a whopping $20,000, the price is right for the target customer, as the BMW i8 starts at $135,700. A price like that kind of makes that luggage set seem like a drop in the bucket.
Image Source
Not only does the luggage fit perfectly size-wise, but its design and appearance fit perfectly with BMW's image: sleek, masculine, and high-quality. Turns out both the luggage and some parts of the car's interior use carbon fiber, strong-yet-light composite material.

Why the Louis Vuitton Co-branding Campaign Works

Both brands knew their target market desired luxury and meticulous craftsmanship. By selling complimentary high-quality products, the brands successfully garnered attention from respective loyal customers.

7. Starbucks & Spotify

Co-branding Campaign: First-of-Its-Kind Music Ecosystem

Starbucks scaled up a premium coffee shop experience into a massive global brand, using music to create an ambiance around its coffee. Spotify, a music streaming platform, has powered almost 25 billion hours of listening worldwide. Starbucks and Spotify forged an innovative co-branding partnership to build a "music ecosystem", offering artists greater access to Starbucks consumers and giving Starbucks access to Spotify's expansive discography.

Why the Spotify Co-branding Campaign Works

Through the initiative, Starbucks employees get a Spotify premium subscription, with which they can curate playlists (that patrons can access through the Starbucks Mobile App) to play throughout the day in the shop. This music ecosystem is designed to expand the coffeehouse environment that Starbucks is known for while giving artists greater exposure to Starbucks customers.

The "musical-ecosystem" partnership is mutually beneficial, allowing the companies to reach the other's audience without sacrificing their brand.
Image Source
8. Apple & MasterCard

Co-branding Campaign: Apple Pay

Sometimes, co-branding partnerships aren't just cool projects between two companies - they actually have practical value when the companies work together.

When Apple released the Apple Pay app, the brand effectively changed how people perform transactions. This app allows people to store their credit or debit card data on their phones, so they can use them without physically having the card with them. But for this app to succeed, it needs credit card companies to integrate with this technology. By the same token, credit card companies also face more competition if they aren't compatible with the latest consumer purchasing tool.

Why the Mastercard Co-branding Campaign Works

To get out ahead of its competition, MasterCard became the first credit card company to allow its users to store their credit and debit cards on Apple Pay. This relationship continued to be useful for both Apple and Mastercard with the release of the Apple Card. MasterCard not only shows continuous support of a major consumer tech developer in this partnership - it evolves with its customers in how they choose to make purchases.

9. Airbnb & Flipboard
Image Source
Co-branding Campaign: Experiences

You've probably heard of Airbnb, the room-sharing application that allows you to find convenient lodging hosted by real people. But its partner, Flipboard, might not be on your radar.

Flipboard is a news aggregator that collects news and topical content that users are sharing on social media, and allows you to "flip" through the material much like a social media feed. Well, Airbnb teamed up with Flipboard to create Experiences, which serve Airbnb users with lifestyle content tailored to their interests and shared by people with similar interests as the reader.

Why the Flipboard Co-branding Campaign Works

The ongoing campaign recently led to another co-created product called Trips, which allows Airbnb users to connect with hosts with common interests and actually book these experiences when traveling. This partnership is an impressive example of how businesses can connect their customers with information that caters to their individual interests and drive product usage as a result.

10. Uber & Spotify

Co-branding Campaign: Soundtrack for Your Ride

Music-streaming app Spotify partnered with ride-hailing app Uber to create "a soundtrack for your ride." This is a great example of a co-branding partnership between two very different products with similar goals- to earn more users.
Image Source
When riders are waiting for an Uber ride, they're prompted to connect with Spotify and become the DJ of their trip. Users can choose from their own playlists to determine what they'll listen to.

Why the Spotify Co-branding Campaign Works

This smart co-branding partnership helps fans of Uber and Spotify alike enjoy better experiences thanks to the app. And they might be more interested in picking Uber and Spotify over competitors knowing they can enjoy their next ride listening to their favorite tunes.

11. Levi's & Pinterest

Co-branding Campaign: Styled by Levi's

Levi Strauss & Co. - one of the world's oldest and most recognized jean brands - recently joined forces with Pinterest, a social platform where users pin posts they like to their user profiles.

People often turn to Pinterest for fashion inspiration, making a co-branding partnership with Levi's a natural partnership. Styled by Levi's is a new initiative between Pinterest and Levi's that offers a "personalized styling experience," or style insights tailored to each user's tastes and preferences.

Why the Pinterest Co-branding Campaign Works

The partnership offers an authentic and individualized brand relationship, which is difficult to come by in an increasingly digital environment. Pinterest offers Levi's a leading social platform with millions of users interested in visual offerings, and Levi's meets these needs with digital personalization and visual-focused boards.

12. BuzzFeed & Best Friends Animal Society

Co-branding Campaign: Emma Watson + Kittens

Some co-branding campaigns are more complicated than others. This example from BuzzFeed and Best Friends Animal Society is one of the simplest ones-and it shows that a great co-branding effort doesn't have to take months of planning or millions of dollars.
Image Source
For this campaign, Best Friends Animal Society wanted to leverage BuzzFeed's readership of over 110 million people.

To do this, they partnered with the folks at BuzzFeed to set up and publish an article called, "We Interviewed Emma Watson While She Played With Kittens And It Was Absolutely Adorable," which you can read here.

The article is exactly what it sounds like: Harry Potter and Beauty and the Beast star Emma Watson answered fans' questions while she played with cute kittens.

Why the Best Friends Animal Society Co-branding Campaign Works

The article ends with a CTA advertising that the kittens featured in the video are, in fact, adoptable - a win-win for both partners by using celebrity to garner attention for clicks while providing more exposure for animals in need of new homes and donations.

13. CoverGirl & Lucasfilm

Co-branding Campaign: Light Side and Dark Side Makeup

Whenever a new installment of the beloved "Star Wars" series is released in theaters, it causes global pandemonium. The release of "Star Wars: Episode VII The Force Awakens" in 2015 was no exception. The series' parent company, Lucasfilm, partnered with CoverGirl to capture a broader audience to get new and old fans excited about the movie's release.
Image Source
You might be wondering, "What do "Star Wars" and makeup have in common?" And the cleverness of this partnership is evident in the answer.

In the past, space-age action movies were almost exclusively advertised and targeted toward men and boys. But today, that's nonsense - because people of all genders can be interested in space exploration and makeup contouring alike.

The line was designed by famed makeup artist Pat McGrath, and it features two styles: the Light Side and the Dark Side, which loyal "Star Wars" fans will recognize as the sides of good and evil in the movies.

Why the Lucasfilm Co-branding Campaign Works

This co-branding partnership was a win for both brands. Lucasfilm captured more attention and got CoverGirl shoppers (many of whom are young women) excited about the film's release. And CoverGirl hopped on the "Star Wars" advertising bandwagon that took over the internet, stores, and TV leading up to the film's release.

14. Amazon & American Express

Co-branding Campaign: Amazon Business American Express Card
Image Source
Ecommerce giant Amazon is a global enterprise with millions of users and almost two million businesses that sell on their platform. Amazon is looking to improve the way small businesses sell on their platform, so they've partnered with American Express on a co-branded credit card.

Why the American Express Co-branding Campaign Works

The card helped users buy goods and services and offered enhanced data insights on their purchasing activity. American Express and Amazon share a commitment to help small businesses grow in the U.S. and by combining their efforts, the two companies had the opportunity to enhance their performance while building brand trust.

15. UNICEF & Target

Co-branding Campaign: Kid Power

If you have the chance to partner for a not-for-profit cause, it can pay off in multiple ways.

Target partnered with UNICEF on a campaign called Kid Power, which committed Target to one of UNICEF's sustainable development goals (SDGs). The retailer sold kid-friendly fitness trackers encouraging the wearer to complete various fitness activities, which ultimately helped deliver food packets to underprivileged children worldwide.
Image Source
Why the Target Co-branding Campaign Works

By selling this fun, inexpensive fitness product, Target encourages children to embrace a healthy lifestyle and uses kids' successes to supply underserved communities with the necessary resources. It's an ongoing partnership that generates awareness of global malnutrition, helps UNICEF meet its demanding SDGs, and opens up Target to a demographic of families they might otherwise have had access to.

16. Nike & Apple

Co-branding Campaign: Apple Watch Nike

Athletic brand Nike and technology giant Apple have been working together since the early 2000s, when the first line of iPods was released.

The co-branding partnership started as a way to bring music from Apple to Nike customers' workouts. Today, Nike Run Club connects Apple Watch users to training, expert coaching, and custom bands that connect runners to a community with the latest technology.

Why the Apple Co-branding Campaign Works
Image Source
It's a genius co-branding move that helps both parties give a better experience to customers - and with the popularity of fitness tracking technology, Nike is ahead of the curve by making it easy for athletes to track while they play.

17. Bonne Belle & Dr. Pepper

Co-branding Campaign: Flavored Lip Balm

Dr. Pepper-flavored lip balm. I mean, it's genius.

Bonne Belle first debuted Lip Smacker, the world's first flavored lip balm, in 1973, starting with flavors like strawberry, lemon, and green apple. Just two years later in 1975, the brand forged its first flavor partnership with the timeless Dr. Pepper brand. The result? A lip balm flavor that's been famous for decades among teenage girls.

If you think the connection between lip balm and Dr. Pepper is a little thin, consider the copy on one of their vintage ads: "It's the super shiny lip gloss with lip-smacking flavor... just like the world's most original soft drink." And later, "From Bonne Belle of course: the cosmetics company that understands your taste."
Image Source
Why the Dr. Pepper Co-branding Campaign Works

While you don't think of flavor and cosmetics, this campaign works because overlapping target demographic consumers can get excited to try an affordable and fun product like Dr. Pepper lip gloss with ease.

Most Surprising Brand Partnerships

Every so often, we hear about a brand partnership that we think doesn't make any sense - but it leads to a few incredibly memorable, unexpected campaigns. If you want to broaden your horizons of who you should partner with, check out these surprising co-marketing successes.

18. Burger King & McDonald's

Co-branding Campaign: A Day Without a Whopper

Yes. You read that right: Burger King and McDonald's - two of the fiercest fast-food industry opponents- joined together for a co-marketing campaign in 2019. And, while we don't encourage competing brands to do this - the campaign was for an amazing cause.

At the time, McDonald's was donating $2 to childhood cancer charities for every Big Mac purchased. So, Burger King worked with the competing fast-food chain to host "A Day Without a Whopper."

For one day, Burger Kings across Argentina and other countries took the Whopper off their menu to encourage people to buy a Big Mac from McDonald's instead.
Image Source
Why the McDonald's Co-branding Campaign Works

Although there were some criticisms of the campaign, because it confused Burger King customers, it was an amazing example of two competing brands that worked together to help a charity. Because of how they partnered up for a good cause, audiences loved the campaign and buzzed about it all over the web.

19. Warby Parker & Arby's

Co-branding Campaign: The WArby Collection

Warby Parker is known for selling affordable glasses at stores across the U.S. and through nearly fully virtual fitting platforms. Arby's is a restaurant chain known for its slogan, "We have the meat!"

Looking at the two companies, there's not much that they have in common. But, this didn't stop them from creating a co-branded April Fools Day campaign in 2018.

For the campaign, the two brands played up the one major thing they had in common - the fact that the word "Arby" was part of both their names.

On April Fool's Day, Warby Parker visitors could buy Arby's themed fashion items, which were part of the cleverly-named WArby Collection.

According to Nylon, items visitors could buy included a WArby's T-shirt, baseball cap, and tote bag (which also said, "Nice To Meat You" on it). They could also buy a pack of sandwich-themed microfiber cloths and glasses with a raw beef pattern.

Meanwhile, Arby's offered food with boxes that had the WArby's logo on it as well as onion rings with a box that read "Onion Ring Monocle."

Why Arby's Co-branding Campaign Works
Image Source
This campaign shows that you don't necessarily need a matching audience demographic or a partner in a similar industry to create a campaign that catches a large audience's attention. Sometimes all you need is a sense of humor and creativity.

20. Taco Bell & Doritos

Co-branding Campaign: Doritos Locos Tacos

When the Doritos Locos Taco was first introduced, it quickly became one of the fast food industry's most popular and widely recognized items. Frito-Lay took Taco Bell's crunchy taco recipe and gave the Locos Taco its special, signature twist: a Doritos shell. The two companies wanted to keep the shell as close to the original cheesy chips as possible, using the original corn masa recipe and coating it in that distinctive nacho cheese dust.

The taco was an immediate and explosive success: Taco Bell sold an estimated 1 billion units the first year it was introduced.

Why the Doritos Co-branding Campaign Works

Taco Bell and Frito Lay advertised the Doritos Locos Taco by wrapping the taco in a classic Doritos bag, reflecting the co-branding partnership and appealing to both brands' audiences. The Doritos Locos Taco's extraordinary success is yet another example of why co-branding can boost reach and sales for both brands.

21. Tim and Eric & Purple

Co-branding Campaign: Purple Boys

Tim and Eric (a.k.a. Tim Heidecker and Eric Wareheim) are the comedic duo behind Adult Swim shows like Tim and Eric Awesome Show, Great Job!, Check It Out! with Dr. Steve Brule, Tim & Eric's Bedtime Stories, and Beef House.

Meanwhile, Purple, known for the popular Purple Mattress, often launches unique advertising campaigns which have caught the eye of millennial and Gen Z demographics.

It makes sense for a company to team up with celebrities who can promote their product, Tim and Eric's comedic style is incredibly random and unique. Although they're pretty hilarious, as a marketer, you might wonder how this type of comedy style translates into a mattress marketing campaign that actually generates revenue.

While you might think Tim and Eric had to dull down their iconic routine to sell Purple products, this is actually quite far from the truth. In a video series called The Purple Boys, they actually amped up their bizarre antics.

The storylines of Purple Boys episodes often relate to a sleep-related pain point, such as the "Sunday Scaries." Although the script and special effects are incredibly random, the episodes almost always end with the Purple Boys supplying a sleep-deprived character with a Purple mattress.

To give you a visual of what this off-the-wall comedy series looks like, check out this episode that centers on a sleepy character named Joe Dunder:

Why the Purple Co-branding Campaign Works

Although hiring actors is technically more of a sponsorship than a partnership, this large-scale campaign benefits both parties. While Tim and Eric could spread awareness of their comedy to Purple audiences, Tim and Eric fans might watch these videos for entertainment and learn about Purple's products along the way.
Image Source
How to Measure the Success of a Co-branding Business Partnership

According to 2022 Foundry Co. research, 56% of those surveyed said their partner marketing programs offered value in the last year.

But if you're new to co-branding, how do you measure that success? There are many ways to quantify partnership success. To begin, think about your co-branding partnerships in three stages.

First, have you found the right co-branding partner? Brand compatibility is key. Then, you need to work out how your brands will collaborate.

Consistent messaging, clear communication between partners, and resource distribution will all impact the quality of your combined efforts. This guide to co-marketing can help you get your first co-branding partnership going.

Next, you'll want to measure the outcomes of your shared efforts. To track a co-branding campaign:

1. Find the Right Metrics

Choose KPIs that both brands track and have easy access to. Be sure that these metrics align with business goals. You'll want KPIs that you can track to measure specific campaigns, as well as KPIs you can track for the life of your partnership. These might include:

ROI

Revenue growth

Customer retention

Website traffic

Social media engagement

Share of voice

2. Limit Your KPIs

Combining the processes of two different teams can get complicated. To make measuring success possible for both teams, be thoughtful in KPI selection, and resist adding new KPIs for each campaign and project. Try to focus on metrics that tie directly to partnership success.

3. Align Tracking Habits

Create a structure for data tracking that lets you compare results with your partner directly. For example, if they're tracking views monthly, and you're tracking weekly, there will be data mismatches that can make things more complex than they need to be.

KPI alignment can help give you a clear picture of your results before the partnership and to track the influence of your partnership over time. This can also help you target where you can improve your co-branding initiatives.

4. Watch the Big Picture

Be sure to track ROI or another KPI like revenue growth to track the overall impact of your new partnership. As you and your partner's needs and strategies shift, this data offers insight into the long-term value of your efforts.

Finally, to gauge the success of your co-branding partnership, set aside time to do an annual review of the partnership. You'll want to check for consistent brand alignment and balanced priorities. Ask yourself:

Are you running the same campaigns over and over?

Have results plateaued?

Are you still coming up with exciting ideas together?

If the answer is no to any of the above, you might need to assess the partnership.

It's also important to check if your co-branding partnership is continuing to meet stakeholder expectations. If either team isn't getting their needs met, you may need to revisit your partnership.

This review is also an exciting time to see how changes in your industry could boost your co-branding work. Set aside time to talk to your partner about new strategies and products that might spark even more exciting collaboration.

These strategies for measuring success can help you make sure that your co-branding start isn't a one-hit-wonder. Instead, it can be a lasting source of value for your business.

Creating a Great Co-marketing Campaign

Although you might not have the budget of the brands noted above, you can take note of their creativity, level of storytelling, and how they pooled their resources to connect their separate brands in a mutually beneficial way.

As you search for the best co-marketing partner for your brand and build a winning campaign, we hope you find some inspiration for your next stellar launch.
Editor's Note: This article was originally published in September 2020 and has been updated for comprehensiveness.

# data[].attributes.attachment_link.title
21 Examples of Successful Co-Branding Partnerships (And Why They're So Effective)

# data[].attributes.attachment_link.description
Need inspiration for your next brand partnership? Check out these real-world examples of great co-branding partnerships - and what makes them so good.

# data[].attributes.body
Tesco allows relatively junior analysts at its corporate headquarters to conduct experiments on small numbers of customers. These employees deliver something that the senior managers generally don’t: a steady stream of creative new ideas that are relevant to younger customers.

# data[].attributes.headline
Before running experiments on anything with high stakes, try designing and running a simple A/B test from scratch in an environment that you entirely control.

# data[].attributes.subject
Before running experiments on anything with high stakes, try designing and running a simple A/B test from scratch in an environment that you entirely control.

# data[].attributes.headline
Imagine you're the CEO of an organisation where teams are not feeling the need to experiment.

# data[].attributes.subject
Imagine you're the CEO of an organisation where teams are not feeling the need to experiment.

# data[].attributes.body
You realise this is putting your company behind - because your competitors keep launching solutions and keeping their customers happier on a quicker pace than yours.

# data[].attributes.headline
Come with a small experiment that can be done in the next 2 months!

# data[].attributes.subject
Come with a small experiment that can be done in the next 2 months!

# data[].attributes.body
Reflect on a barrier that you would like to tackle and think of a small experiment that you can do to tackle it.

# data[].attributes.body
Pharmaceutical company Roche throws a champagne brunch after a promising compound fails in testing. Their goal is to sustain team’s creativity and dedication to innovation.

Company: Roche
Intention: Create a culture of iteration and controlled high-risk

# data[].attributes.attachment_link.readable_text
Apple and Nike partnered in September to bring runners the Apple Watch Nike+, which combines Nike Sports Bands with Apple Watch Series 2.

Now, the partnership is continuing with NikeLab - the company's collaborative innovation platform - with the launch of the limited edition Apple Watch NikeLab.
Get The Latest Sports Tech News In Your Inbox!
The neutral-toned Apple Watch NikeLab features a new strap design, integration with the Nike+ Run Club app, exclusive Siri commands, GPS, two-times-brighter display and water resistance to 50 meters all powered by a dual-core processor and watchOS 3.

The watch was made available beginning Thursday on nike.com, at NikeLab doors and the Apple Watch Store at Isetan Shinjuku.

# data[].attributes.attachment_link.title
Nike, Apple Continue Partnership With Release Of Apple Watch NikeLab

# data[].attributes.attachment_link.description
Apple and Nike partnered in September to bring runners the Apple Watch Nike+, which combines Nike Sports Bands with Apple Watch Series 2. Now, the partnership is continuing with NikeLab - the company's collaborative innovation platform - with the launch of the limited edition Apple Watch NikeLab.

# data[].attributes.headline
Look for experiments that are easy to execute using existing resources and staff

# data[].attributes.subject
Look for experiments that are easy to execute using existing resources and staff

# data[].attributes.body
Experiments depend on measurement. If you can’t properly measure attribution from a digital ad to a sale, for example, you’ll have no luck running an experiment to figure out which ads are actually effective.

# data[].attributes.headline
In this page we have put together some ideas to inspire you!

# data[].attributes.subject
In this page we have put together some ideas to inspire you!

# data[].attributes.body
You have 15 minutes to complete this task, and select a team member to present your idea back.

# data[].attributes.attachment_link.readable_text
In 1974, 3M scientist Art Fry came up with a clever invention. He thought if he could apply an adhesive (dreamed up by colleague Spencer Silver several years earlier) to the back of a piece of paper, he could create the perfect bookmark, one that kept place in his church hymnal. He called it the Post-It Note.

What you might not know is that Fry came up with the now iconic product (he talks to the Smithsonian about it here) during his "15 percent time," a program at 3M that allows employees to use a portion of their paid time to chase rainbows and hatch their own ideas. It might seem like a squishy employee benefit. But the time has actually produced many of the company's best-selling products and has set a precedent for some of the top technology companies of the day, like Google and Hewlett-Packard.

The 15 percent program seems squishy, is now key to 3M's business strategy.

Today, 3M is a multinational powerhouse, with more than $20 billion in annual sales across a product line 50,000 deep, from adhesives to optical film. It boasts 22,800 patents, many derived from its 15 percent program. The program has been key to 3M's business strategy and could be a model for other companies eager to innovate. Says Kurt Beinlich, a technical director for 3M: "It's really shaped what and who 3M is."

Founded in 1902 in a little town on the shores of Lake Superior, 3M started out in the mining business as the Minnesota Mining and Manufacturing Company. The company thought it had found corundum, a mineral ideal for making sandpaper. But instead, it was low-grade anorthosite - of little value. With mining hopes dashed, the founders bought a sandpaper factory and struggled for years over how to run it. New investors had to pour in cash to keep it afloat. Eventually, one of them, Lucius Ordway, moved the company to St. Paul, where 3M hit upon some key inventions, among them: masking tape and cellophane tape.

3M launched the 15 percent program in 1948. If it seems radical now, think of how it played as post-war America was suiting up and going to the office, with rigid hierarchies and increasingly defined work and home roles. But it was also a logical next step. All those early years in the red taught 3M a key lesson: Innovate or die, an ethos the company has carried dutifully into the 21st century.

15 percent time is extended to everyone. Who knows who'll create the next Post-It Note?

"It's one of the things that sets 3M apart as an innovative company, by sticking to that culture of giving every one of our employees the ability to follow their instincts to take advantage of opportunities for the company," says Beinlich, who tries to get most of his 70-person technical lab team to participate.

How is the program implemented? In Beinlich's telling, workers often use 15 percent time to pursue something they discovered through the usual course of work but didn't have time to follow up on. And even that depends on other factors - how closely managers keep tabs on projects, for one. What's more, 15 percent time is extended to everyone, not just the scientists (you can hear the cheers in marketing), the idea being: Who knows where the next Post-It Note will come from?

There is failure. As a company culture, it's accepted, if not entirely embraced. In Beinlich's department, engineers designed a heat-repelling cover to protect car finishes from welding sparks. But there just wasn't a market for it: Automotive workers didn't want to shell out for another product when they could keep layering blankets to protect finishes like they always had. "When we found that out, we celebrated that we had found something that was innovative and had its place. But we said OK; let's move on,?" Beinlich says.

The 15 percent program has clearly inspired copycats. Google's 20 percent time famously gave birth to Gmail, Google Earth, and Gmail Labs. (Google would neither confirm nor deny that the idea for its program came from 3M, but it's hard to imagine otherwise; after all, 3M's program had been around 50 years before Google even filed incorporation papers.) Likewise, Hewlett-Packard Labs offers personal creative time.

Still, it's a rare perk at most companies, technical or not. For starters, it's expensive. 3M invests more than $1 billion in R&D alone; 15 percent of that starts to be a sizable outlay. Author Scott Berkun writes about business innovation. He says these policies only work when the outcomes are backed. "Many companies have tried to emulate the ?20 percent time idea" but failed because they remained conservative about supporting the new ideas,? he says. And experts agree that this kind of nudging probably works best at companies where there's a high level of creative competitiveness; that is, where impressing peers is just as important as the innovation itself.

Some have tried to emulate 3M's program but failed because they wouldn't support the new ideas.

3M's got that in spades. Once a year, about 200 employees from dozens of divisions make cardboard posters describing their 15 percent time project as if they were presenting volcano models at a middle school science fair. They stand up their poster, then hang out next to it, awaiting feedback, suggestions, and potential co-collaborators. Wayne Maurer is an R&D manager in 3M's abrasives division and calls it a chance for people to unhinge their "inner geek." He elaborates: "For technical people, it's the most passionate and engaged event we have at 3M."

Past projects have included making clear bandages, optical films that reflect light (seen above), and designing a way to make painter's tape stick to wall edges (to protect against paint bleed). All these products are on the market now.

Sometimes ideas can languish for years. One worker had a hunch that if he reshaped particles on sandpaper, they wouldn't dull so quickly. But that was 15 years ago, and the technology and feedback weren't there to advance it beyond an interesting idea. Two years ago, the same worker started looking at the problem again during his 15 percent time. He made a poster. This time, he got different feedback with the help of new employees and new technology. They discovered they could retain a particle's sharp, pyramid-like shape just by changing the mixing order (see image up top). Now 3M has a winner in the Cubitron II (above), a sandpaper that acts more like a cutting tool -? and one that still stumps copycats, despite that it's been on the market since 2009. If not for the 15 percent time, this worker's idea might've never taken off.

Another obvious benefit of this "think time" is in recruiting. Specialized workers are highly prized and fought for. Companies that offer roughly the same salary as another, can tip the scales with paid personal time. (The snow in Minnesota might be another issue.)

"What you're offering is essentially freedom, and that is very attractive for the right person," says Henry Chesbrough, a professor at the Haas School of Business at UC Berkeley, and the father of open innovation business practices.

Paid personal time is, of course, just one way to help a company innovate and, given the expense, it's not best for everyone. Chesbrough says advances in technology can be achieved if companies generally soften boundaries between where ideas come from and how they take root. A company can limit risk by letting internal ideas spin off into external companies, which might be bought back. On the flip side, internal groups can pursue external ideas. Perhaps the real lesson is that the best ideas can come from anywhere. And an innovative company will find a way to champion them.

# data[].attributes.attachment_link.title
How 3M Gave Everyone Days Off and Created an Innovation Dynamo

# data[].attributes.attachment_link.description
In 1974, 3M scientist Art Fry came up with a clever invention. He thought if he could apply an adhesive (dreamed up by colleague Spencer Silver several years earlier) to the back of a piece of paper, he could create the perfect bookmark, one that kept place in his church hymnal.

# data[].attributes.body
When UBS was considering how to use experiments to improve its wealth management business, it recognized that the place to start was customer acquisition, not improving lifetime customer value.

The effects of experiments on
customer acquisitions can be measured immediately, while the impact on customer lifetime value could take 25 years to assess.

# data[].attributes.body
They reportedly discovered that it was profitable to send coupons for organic food to customers who bought wild birdseed. This is out-of-the-box thinking.

# data[].attributes.headline
Much of what companies learn from mammoth experiments can be gleaned from smaller tests that involve fewer variables, saving resources for follow-up tests.

# data[].attributes.subject
Much of what companies learn from mammoth experiments can be gleaned from smaller tests that involve fewer variables, saving resources for follow-up tests.

# data[].attributes.headline
Since 2009, coders have created thousands of amazing experiments using Chrome, Android, AI, Web VR, AR and more at Google

# data[].attributes.subject
Since 2009, coders have created thousands of amazing experiments using Chrome, Android, AI, Web VR, AR and more at Google

# data[].attributes.body
In Booking.com we split the audience into two groups. One half keeps the current version of a product, while the other half sees an updated version.

We compare behaviours and if those using the new version behave as anticipated then we conclude the experiment worked as expected and put the change ‘live’ for everyone else.

# data[].attributes.body
“It’s one of the things that sets 3M apart as an innovative company, by sticking to that culture of giving every one of our employees the ability to follow their instincts to take advantage of opportunities for the company,” says Beinlich, who tries to get most of his 70-person technical lab team to participate"

# data[].attributes.attachment_link.readable_text
Since 2009, coders have created thousands of amazing experiments using Chrome, Android, AI, AR and more. We're showcasing projects here, along with helpful tools and resources, to inspire others to create new experiments. Here are collections of experiments to explore, with new ones added every week. Have fun.

# data[].attributes.attachment_link.description
Since 2009, coders have created thousands of amazing experiments using Chrome, Android, AI, WebVR, AR and more. We're showcasing projects here, along with helpful tools and resources, to inspire others to create new experiments.

# data[].attributes.attachment_link.readable_text
In a constantly changing industry, evolution of products and services is essential. Click. caught up with Lukas Vermeer, Director of Experimentation at Booking.com to discover how Booking.com uses controlled experimentation to conduct rapid, evidence-based product development.
Click.: Tell me about experimentation at Booking.com?
Vermeer: We run in excess of 1,000 concurrent experiments at any given moment across different products and target groups, allowing us to rapidly validate ideas and implementations. Each test is tailored towards a particular audience because we're targeting specific solutions. This enables us to gather learnings on customer behaviours and discover what makes a meaningful difference for our users.

One way we do this is through A/B testing - something Booking.com is well known for - where we split the target audience into two groups. One half keeps the current version of a product, while the other half sees an updated version. We compare behaviours and if those using the new version behave as anticipated then we conclude the experiment worked as expected and put the change 'live' for everyone else. If it doesn't work as assumed, then we've either misunderstood the problem or what the solution should be. At the very least, we've learned something and ensured we're not making a change that doesn't have the impact we want.

We don't experiment because we like running experiments, but because experimentation is a great way to make sure that when we think we're fixing something, we're actually fixing it. Change is constant, we have to keep updating our products to make them better, but we also have to make sure those changes really work.
Click.: With so many experiments running at once, is there a risk of them overlapping?
Vermeer: Experiments can overlap, yes. In technical terms, you would say there's a risk of interaction effects - two experiments having a different effect in isolation than when they are combined - but the ability to detect when this happens is actually an important benefit of experimentation. In product development it's much more dangerous to isolate the experiments than to run them overlapping; you could test two changes that work independently, but if you isolate the tests you've no idea what's going to happen when both are put live. We realise our overlapping approach has limits, but the intent is to build a better product, not perfect science.
Click.: What's the importance of businesses challenging their assumptions?
Vermeer: If we make changes based purely on assumptions, we risk making unnecessary adjustments to our products. It's crucial to check whether our changes actually have the desired effect on peoples' behaviour. The worst thing we could do is constantly make all of these updates to our products, potentially confusing users if the changes don't actually work how we intended.

One of the most important aspects of the entire scientific method and of experimentation is that you write assumptions down. There's a practice we apply called pre-registration, where experiment owners specify from the start the assumptions they are making, how they are going to impact the customer behaviour and the metrics they will use to measure the experiment.
Click.: How can businesses empower employees to experiment?
Vermeer: This approach has always been ingrained in our company culture, with almost every change we make - no matter how big or small - wrapped in an experiment. I think two of the most important ingredients are transparency and accountability. We really encourage peer reviews - there needs to be some way for people to see each other's work and give their input. We've built this into our infrastructure by making descriptions of all iterations available, with the platform acting as a searchable repository of all previous experiments.

It's also important for leaders to know that it's okay to fail and be wrong - and communicate this to staff. When it comes to experimentation, people can sometimes be hesitant to adopt these methodologies. In my opinion, this is not a technical problem, but rather a mindset: you have to be willing to admit you don't always know what's best - and that's okay. You need to be a little bit vulnerable and open to changing your mind. If instead you have a very set vision for what is right, and nothing could ever challenge the assumptions and opinions underpinning that vision, then effectively using experimentation is going to be problematic.

# data[].attributes.attachment_link.title
The role of experimentation at Booking.com

# data[].attributes.attachment_link.description
In a constantly changing industry, evolution of products and services is essential. Click. caught up with Lukas Vermeer, Director of Experimentation at Booking.com to discover how Booking.com uses controlled experimentation to conduct rapid, evidence-based product development. Click.: Tell me about experimentation at Booking.com?

# data[].attributes.headline
Set a grand challenge that can be broken into testable hypotheses and key performance metrics.

# data[].attributes.subject
Set a grand challenge that can be broken into testable hypotheses and key performance metrics.

# data[].attributes.body
Employees need to see how their experiments support an overall strategic goal. Say Booking.com’s senior leaders challenged employees to design the best online experience in the industry. They might expect that a superior experience would generate more customer traffic, which would attract more suppliers to Booking.com’s platform, helping expand the customer base and activity even more. To discover ways to pursue that goal, employees could devise hypotheses and related metrics—for instance, that underlining important text would increase conversion rates by making critical information easier to find, and that a “one click, no cost” cancellation option would boost user return rates without causing net hotel bookings to drop.

# data[].attributes.body
Any employee at Booking.com can launch an experiment on millions of customers without management’s permission. About 75% of its 1,800 technology and product staffers actively use the company’s experimentation platform. Standard templates allow them to set up tests with minimal effort, and processes like user recruitment, randomization, the recording of visitors’ behavior, and reporting are automated. A core experimentation team and five satellite teams used to provide training and support to the whole organization, but because the firm’s needs evolved, that structure was recently changed to four central teams that report to Vermeer and specialists (“ambassadors”) that are placed in product teams.

# data[].attributes.body
"Leaders have to live by the same rules as everyone else and subject their own ideas to tests. You can’t have an ego, thinking that you always know best. If I, as the CEO, say to someone, ‘This is what I want you to do because I think it’s good for our business,’ employees would literally look at me and say, ‘OK, that’s fine, we are going to test it and see if you are right.’” Bosses ought to display intellectual humility and be unafraid to admit, “I don’t know.” They should heed the advice of Francis Bacon, the father of the scientific method: “If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.” - Gillian Tans

# data[].attributes.attachment_link.readable_text
Idea in Brief

The Opportunity

In an increasingly digital world, randomized, controlled A/B experiments are an extremely valuable way to create or improve online experiences.

The Obstacle

Culture-not tools and technology-prevents companies from conducting the hundreds, even thousands, of tests they should be doing annually and then applying the results.

The Remedy

Create an environment in which curiosity is nurtured, data trumps opinion, anyone can conduct a test, all experiments are done ethically, and managers embrace a new model of leadership.

In December 2017, just before the busy holiday travel season, Booking.com's director of design proposed a radical experiment: testing an entirely new layout for the company's home page. Instead of offering lots of options for hotels, vacation rentals, and travel deals, as the existing home page did, the new one would just feature a small window asking where the customer was going, the dates, and the number of people in the party, and present three simple options: "accommodations," "flights," and "rental cars." All the content and design elements-pictures, text, buttons, and messages-that Booking.com had spent years optimizing would be eliminated.

Gillian Tans, Booking.com's CEO at the time, was skeptical. She worried that the change would cause confusion among the company's loyal customers. Lukas Vermeer, then the head of the firm's core experimentation team, bet a bottle of champagne that the test would "tank"-meaning it would drive down the company's critical performance metric: customer conversion, or how many website visitors made a booking. Given that pessimism, why didn't senior management just veto the trial? Because doing so would have violated one of Booking.com's core tenets: Anyone at the company can test anything-without management's permission.

Booking.com runs more than 1,000 rigorous tests simultaneously and, by my estimates, more than 25,000 tests a year. At any given time, quadrillions (millions of billions) of landing-page permutations are live, meaning two customers in the same location are unlikely to see the same version. All this experimentation has helped transform the company from a small Dutch start-up to the world's largest online accommodation platform in less than two decades.

Booking.com isn't the only firm to discover the power of online experiments. Digital giants such as Amazon, Facebook, Google, and Microsoft have found them to be a game changer when it comes to marketing and innovation. They've helped Microsoft's Bing unit, for instance, make dozens of monthly improvements, which collectively have boosted revenue per search by 10% to 25% a year. (See "The Surprising Power of Online Experiments," HBR, September-October 2017.) Firms without digital roots-including FedEx, State Farm, and H&M-have also embraced online testing, using it to identify the best digital touchpoints, design choices, discounts, and product recommendations.

Harold Edgerton ©2010 MIT. Courtesy of MIT Museum
Harold Edgerton was known for his experiments with high-speed photography and used stroboscopic equipment to capture moments in time.

"In an increasingly digital world, if you don't do large-scale experimentation, in the long term-and in many industries the short term-you're dead," Mark Okerstrom, the CEO of Expedia Group told me. "At any one time we're running hundreds, if not thousands, of concurrent experiments, involving millions of visitors. Because of this, we don't have to guess what customers want; we have the ability to run the most massive 'customer surveys' that exist, again and again, to have them tell us what they want."

But in studying more than a dozen organizations and analyzing anonymized data on experiments from upwards of 1,000, I have seen that Booking.com, Expedia, and their ilk are the exception. Instead of running hundreds or thousands of online tests a year, many firms run no more than a few dozen that have little impact.

If testing is so valuable, why don't companies do it more? After examining this question for several years, I can tell you that the central reason is culture. As companies try to scale up their online experimentation capacity, they often find that the obstacles are not tools and technology but shared behaviors, beliefs, and values. For every experiment that succeeds, nearly 10 don't-and in the eyes of many organizations that emphasize efficiency, predictability, and "winning," those failures are wasteful.

To successfully innovate, companies need to make experimentation an integral part of everyday life-even when budgets are tight. That means creating an environment where employees' curiosity is nurtured, data trumps opinion, anyone (not just people in R&D) can conduct or commission a test, all experiments are done ethically, and managers embrace a new model of leadership. In this article, I'll look at several companies that have managed to do those things well, focusing in particular on Booking.com, which has one of the strongest cultures of experimentation I have found.

Cultivate Curiosity

Everyone in the organization, from the leadership on down, needs to value surprises, despite the difficulty of assigning a dollar figure to them and the impossibility of predicting when and how often they'll occur. When firms adopt this mindset, curiosity will prevail and people will see failures not as costly mistakes but as opportunities for learning.

A classic example concerns an incident at Amazon involving a revision of Air Patriots, a game for mobile devices in which players defend towers from attack with a squadron of planes. When Amazon launched a new version of it, the development team was taken aback by the response: The seven-day user-retention rate dropped by an astonishing 70%, and revenue fell 30%. The team discovered that it had inadvertently increased the game's difficulty by about 10%. Amazon quickly shipped a fix, but the developers wondered if making the game easier could produce large gains in retention and revenue. To find out, they ran a test with four new levels of difficulty, in addition to a control, and learned that the easiest variant did the best. After some further refinements, Amazon launched a new version-and this time users played 20% longer and revenue increased by 20%. An accident had led to a surprising insight, which became the starting point for new experiments.

Unfortunately, this kind of reaction is an anomaly. At many companies the risk associated with experiments makes managers reluctant to allocate resources to them. But the gains enjoyed by companies that have made the leap of faith should give others the courage to follow them.

Many organizations are also too conservative about the nature and amount of experimentation. Overemphasizing the importance of successful experiments may encourage employees to focus on familiar solutions or those that they already know will work and avoid testing ideas that they fear might fail. And it's actually less risky to run a large number of experiments than a small number. At Booking.com, only about 10% of experiments generate positive results-meaning that "B," a modification that attempts to improve something (sales, repeat usage, click-through rates, or the time users spend on the site, for example), performs better among randomly assigned users than "A," the control, which is the status quo. (In addition to A/B tests, Booking.com also runs more-complex tests that assess more than one modification at the same time.) But when you conduct a large volume of experiments, a low success rate still translates into a significant number of successes, which, in turn, diminish the financial and emotional costs of the failures. If a company does only a handful of experiments a year, it may have only one success or, if it's unlucky, none. Then failure is a big deal.

At the companies I studied, the success rate for ideas tested early in the development of a brand-new offering is even lower. Early failures, however, allow developers to quickly eliminate unfavorable options and refocus their efforts on more-promising alternatives.

In experimental cultures, employees are undaunted by the possibility of failure. "The people who thrive here are curious, open-minded, eager to learn and figure things out, and OK with being proven wrong," said Vermeer, who now oversees all testing at Booking.com. The firm's recruiters look for such people, and to make sure they're empowered to follow their instincts, the company puts new hires through a rigorous onboarding process, which includes experimentation training, and then gives them access to all testing tools.

Insist That Data Trump Opinions

The empirical results of online experiments must prevail when they clash with strong opinions, no matter whose opinions they are. This is the attitude at Booking.com, but it's rare among most firms for an understandable reason: human nature. We tend to happily accept "good" results that confirm our biases but challenge and thoroughly investigate "bad" results that go against our assumptions.

The remedy is to implement the changes that experiments validate with few exceptions. As one director at Booking.com told me, "If the test tells you that the header of the website should be pink, then it should be pink. You always follow the test."

Andreas Feininger/The LIFE Picture Collection via Getty Images
A long-exposure photograph by Andreas Feininger captures the light trail of a helicopter.

Getting executives in the top ranks to abide by this rule isn't easy. (As the American writer Upton Sinclair once quipped, "It is difficult to get a man to understand something, when his salary depends upon his not understanding it!") But it's vital that they do: Nothing stalls innovation faster than a so-called HiPPO-highest-paid person's opinion.

Note that I'm not saying that all management decisions can or should be based on online experiments. Some things are very hard, if not impossible, to conduct tests on-for example, strategic calls on whether to acquire a company.

But if everything that can be tested online is tested, experiments can become instrumental to management decisions and fuel healthy debates. Sometimes, those discussions might result in a conscious choice to overrule the data. That's what happened with one decision involving a comedy series at Netflix, which has built a sophisticated infrastructure for large-scale experimentation. According to a Wall Street Journal article published in 2018, the company's executives were torn when tests showed that a promotion featuring an image of only Lily Tomlin, one of the stars of Grace and Frankie, resulted in more clicks by potential viewers than promotions featuring both Tomlin and her costar, Jane Fonda. The content team worried that excluding Fonda would alienate the actress and possibly violate her contract. After heated debates that pitted empirical evidence against "strategic considerations," Netflix chose to use images that included both actresses, even though customer data didn't support the decision. However, the experimental evidence made the trade-offs more transparent.

Democratize Experimentation

As I've noted, any employee at Booking.com can launch an experiment on millions of customers without management's permission. About 75% of its 1,800 technology and product staffers actively use the company's experimentation platform. Standard templates allow them to set up tests with minimal effort, and processes like user recruitment, randomization, the recording of visitors' behavior, and reporting are automated. A core experimentation team and five satellite teams used to provide training and support to the whole organization, but because the firm's needs evolved, that structure was recently changed to four central teams that report to Vermeer and specialists ("ambassadors") that are placed in product teams.

To get things rolling, individuals or teams fill out an electronic form, which is visible to all and includes the name of the experiment, its purpose, the main beneficiaries (customers or suppliers), related past experiments, and the number of modifications to be tried out in A/B, A/B/C, or A/B/n tests. Once an experiment is up and running, the team watches it closely for the first few hours; if its primary or secondary metrics tank quickly, the team can stop the test. After that initial period, the platform continues to automatically run data-quality checks and sends warning messages if something is odd. To encourage openness, Booking.com maintains a central searchable repository of past experiments, with full descriptions of successes, failures, iterations, and final decisions. And everyone can see the real-time information generated by ongoing experiments.

"Somewhat ironically, the centralizing of our experimentation infrastructure is what makes our organizational decentralization possible," Vermeer explained to me. "Everyone uses the same tools. This fosters trust in each other's data and enables discussion and accountability. While some companies, like Microsoft, Facebook, and Google, may be more technically advanced in areas like machine learning, our use of simple A/B tests makes us more successful in getting all people involved; we have democratized testing throughout the organization."

Democratization, of course, has its challenges. One is the risk that teams or individuals could break something on Booking.com's high-traffic website, causing it to crash. Another is that each team has to set its own direction and figure out which user problems it wants to solve. That requires extensive training and ongoing discussions among team members about what the right problems are. Debates are encouraged, and people reach out to colleagues if they see anything that strikes them as questionable. Just as anyone can launch an experiment, anybody can stop one. However, this happens only on the rare occasion when an experiment has gone catastrophically awry-for example, if someone is alone in the office at night and sees that an experiment is causing a key metric like the customer conversion rate to plunge and will cost the company millions of dollars in revenues if it continues.

This system gives teams the autonomy they need to try out new approaches they believe are valuable and allows people throughout the company to monitor the experiments and provide feedback in real time. It truly liberates everyone to test any idea about how to improve Booking.com's business.

Be Ethically Sensitive

When contemplating new experiments, companies must think carefully about whether users would consider the tests to be unethical. While the answer isn't always clear-cut, organizations that fail to examine this question risk sparking a backlash. Take the weeklong experiment that Facebook ran in 2012 to learn whether emotional states were contagious on its platform. Facebook rejiggered its news feed-an algorithmically curated list of posts, stories, and activities-to see whether viewing fewer positive news stories led people to reduce their number of positive posts. The network also tested whether the reverse happened when people were exposed to fewer negative news stories. The experiment involved nearly 690,000 randomly selected users, about 310,000 of whom were unwittingly exposed to manipulated emotional expressions in their news feeds, while the rest were subjected to control conditions in which a corresponding number of randomly chosen posts were omitted.

When researchers from Facebook and Cornell University published the results in an academic journal, public outrage erupted. Facebook's data science team had been running experiments on unsuspecting users for years without controversy, but the emotional manipulation struck a nerve. Critics raised concerns about whether the participants' consent to Facebook's general data-use policy sufficed; they felt the company should have made it clearer that users could opt out of testing and that data was collected for research. From a learning perspective, the experiment was a success: It found that emotional contagion existed online, though the effect was very small. But some users felt that Facebook had exploited them in the name of science.

Research suggests that companies that test new ideas first face greater customer scrutiny than competitors that implement new practices without conducting any experiments. In a published analysis of 16 studies in domains such as health care, vehicle design, and global poverty, bioethicist Michelle Meyer and her collaborators concluded that participants considered A/B tests to be more morally questionable than the universal implementation of an untested practice (A or B) on the entire population-even when both treatments were unobjectionable.

Clearly, ethics training and some kind of oversight are necessary. The challenge is conducting the latter in ways that don't make people overly cautious or tangle them in red tape. For those precise reasons, Booking.com has shied away from imposing rules from on high about what kind of tests can be run. Instead, it encourages employees to ask whether an experiment or proposed practice would help or hurt customers. "I'd rather stay away from policing or ethical review boards," David Vismans, Booking.com's chief product officer, told me. "That's not a scalable solution. You'd create a bottleneck, and testing police don't make people feel like they're empowered." Instead, the company encourages debates in internal online forums that are open to all employees. The debates can be vigorous and have tackled issues like the use of techniques to persuade customers to complete transactions (for example, messages such as "Please book now or you will lose this reservation" or "Only three rooms left"). "I would rather have a community that is self-correcting," Vismans explained.

To that end, Booking.com's onboarding process also includes ethics training. LinkedIn, another company with a large experimentation program, takes a slightly different approach. It has created internal guidelines that state the company won't run experiments "that are intended to deliver a negative member experience, have a goal of altering members' moods or emotions, or override existing members' settings or choices."

Embrace a Different Leadership Model

By democratizing experimentation and following test results where they lead, companies can enable employees to make good decisions on their own and accelerate innovation and improvements. But if most decisions are made this way, what's left for senior leaders to do, beyond developing the company's strategic direction and tackling big decisions such as which acquisitions to make? There are at least four things:

Set a grand challenge that can be broken into testable hypotheses and key performance metrics.

Employees need to see how their experiments support an overall strategic goal. Say Booking.com's senior leaders challenged employees to design the best online experience in the industry. They might expect that a superior experience would generate more customer traffic, which would attract more suppliers to Booking.com's platform, helping expand the customer base and activity even more. To discover ways to pursue that goal, employees could devise hypotheses and related metrics-for instance, that underlining important text would increase conversion rates by making critical information easier to find, and that a "one click, no cost" cancellation option would boost user return rates without causing net hotel bookings to drop.

Put in place systems, resources, and organizational designs that allow for large-scale experimentation.

Scientifically testing nearly every idea requires infrastructure: instrumentation, data pipelines, and data scientists. Several third-party tools and services make it easy to try experiments, but to scale things up, senior leaders must tightly integrate the testing capability into company processes. Doing so requires striking the right balance between centralization and decentralization.

In centralized groups, dedicated specialists such as developers, user interface designers, and data analysts can run experiments for the entire company and focus on introducing state-of-the-art methods and tools. But if testing is limited to a small group of specialists, it will be hard to scale up experimentation and change a company's culture. In decentralized testing, firms spread specialist teams throughout different business units. While this approach expands experimentation to more parts of the organization, it can hinder knowledge sharing and lead to conflicting goals and poor coordination among specialists. Decentralization may be needed to get the broader organization involved at first, but after that, firms should turn to improving their experimentation capabilities. That's what Booking.com did. It initially used satellite teams to spread experimentation across the company but found that they were too busy supporting users to focus on building firmwide capabilities. To address that problem and align the teams better, Booking.com recently switched to a center-of-excellence model that supports business units, standardizes the company's approach to experimentation, and makes sure that best practices are adopted and followed.

Be a role model.

Leaders have to live by the same rules as everyone else and subject their own ideas to tests. "You can't have an ego, thinking that you always know best," Tans told me. "If I, as the CEO, say to someone, 'This is what I want you to do because I think it's good for our business,' employees would literally look at me and say, 'OK, that's fine, we are going to test it and see if you are right.'" Bosses ought to display intellectual humility and be unafraid to admit, "I don't know." They should heed the advice of Francis Bacon, the father of the scientific method: "If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties."

Recognize that words alone won't change behavior.

Ultimately, being a leader in an experiment-driven organization means letting go and empowering employees to perform their own tests-which doesn't happen by simply telling people that they can do so. It requires a concerted effort like IBM's.

Every day, employees at the company use A/B tests to try out their ideas for tweaks. Below are two examples.

In 2015 experimentation wasn't a core activity at IBM; the company's IT function offered to run tests, but they were costly, were charged back to business units, and had to follow a rigid process. The testing capacity consisted of just one specialist, who was also the gatekeeper and who rejected many proposed experiments because he felt that they weren't strong-enough candidates. As a result, the company ran only 97 tests that year. Then, Ari Sheinkin, IBM's head of marketing analytics at the time, took over experimentation and, with the backing of the chief marketing officer, empowered over 5,500 marketers worldwide to conduct their own tests. To induce them to do so, Sheinkin took a number of steps. He installed easy-to-use tools, created a center of excellence to provide support, introduced a framework for conducting disciplined experiments, offered training for everyone, and made online tests free for all business groups. He also conducted an initial "testing blitz" during which the marketing units had to run a total of 30 online experiments in 30 days. After that, he held quarterly contests for the most innovative or most scalable experiments. He also employed more-forceful tactics: IBM tied part of marketing units' budgets to experimentation plans. These efforts worked. By 2018, the number of annual tests had surged to 2,822.

CONCLUSION

Realizing the transformative power of experimentation requires a sustained commitment. Over time experiments will result in thousands of small and not-so-small changes that collectively generate huge benefits. Providing the right tools, while essential, is the easy part and isn't enough to make experimentation a way of life. Vismans put it best: "If I have any advice for CEOs, it's this: Large-scale testing is not a technical thing; it's a cultural thing that you need to fully embrace. You need to ask yourself two big questions: How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you? And if the answer is that you don't like to be proven wrong and don't want employees to decide the future of your products, it's not going to work. You will never reap the full benefits of experimentation."

The lesson is that it's not so important whether any one experiment succeeds or fails; what matters is how decisions are adjudicated under uncertainty in an organization. They should not be based on faith or personal opinion alone. If they can be put to the test, they should be.

# data[].attributes.attachment_link.description
Idea in Brief The Opportunity In an increasingly digital world, randomized, controlled A/B experiments are an extremely valuable way to create or improve online experiences. The Obstacle Culture-not tools and technology-prevents companies from conducting the hundreds, even thousands, of tests they should be doing annually and then applying the results.

# data[].attributes.headline
SUBMIT YOUR IDEAS ON MENTI.COM (click on the link below) 

# data[].attributes.subject
SUBMIT YOUR IDEAS ON MENTI.COM (click on the link below) 