eCommerce Solutions: Big Data and Walmart Subscription Boxes

Scandiweb recently came on board as a custom eCommerce solution developer for a Walmart affiliate specialized in branding and targeted audience reach. The company connects major brands in 14 lifestyle segments with their target audience through engaging branding campaigns. This includes sending out free samples, adding free tryouts for new products to customers’ purchases, and, since 2014, online subscription boxes comprised of brand products as part of the Walmart eCommerce initiative.

Going from Offline Branding to an Online eCommerce Solution

In the traditional offline model, pinpointing the target audience was easy. When a customer bought a product, you could be fairly certain that they would be interested to try out a sample of a similar product. The model is incredibly effective at attracting new customers — the company in question boasts a 97+ percent try out rate, with subsequent conversion rate at 14–32 percent.

Migrating the model to the online marketplace, however, has required some innovative techniques. The subscription box service that Scandiweb has been working on has been available to the public since 2014. It offers two product options: trendsetter (for persons at the age of 35 and younger) and classic (for those older than 35). With a subscription cost of $5 per package, each box contains products valued between $10 and $15.

With a selection of brand products delivered right to your home for a fraction of the retail price, it’s no wonder that the subscription box business is booming. The ever-increasing competition, however, is yet another reason to amp up the company’s offer.

It’s safe to say that dividing all the customers in two groups based solely on age is not optimal. When attracting new customers online, there is no existing purchase history that can be used to categorize the client. Thus, a definitive, yet unobstructive client grouping method is needed to ensure that the end customer receives products they’re truly interested in.

eCommerce Solution for Customer Segmentation – Big Data

A new model was developed by Scandiweb. Its main driving force is a customer survey, which segments according to answers they’ve given to a few easy questions, and adds more options for subscription product customization, and enables more targeted customer segmentation.

As prospects answer to such questions “What’s your skin type?” or “Which of these brands do you like?”, they can easily be grouped on a much more detailed scale than before. From there, the data can be used to assort the best subscription boxes with the brands and products that the client is known to have interest in.

For the model to work, the survey must be completed by all clients. For new customers, this means that the survey is a mandatory step before being able to make a purchase. For existing clients, the survey will be presented before subscription renewal is possible.

From this perspective, the new model lets the company hit two birds with one stone. First, the end customer receives a more targeted package, and their satisfaction is likely to go up. And, secondly, the company can negotiate with specific major brands about promoting their new products via subscription boxes to the perfect target audience, pinpointed with laser precision.

In the future, feedback about the received products will be automatically integrated into Walmart’s product view. This way, the products sent out via subscription boxes will aggregate customer feedback even before they’re launched in retail.

However, the real beauty of the model lies in its scalability. New surveys can be added seamlessly, gathering even more data on user interests and preferences. The data is easily exported, which then allows for advanced categorization and fully customized box assortment in the future.

Behind the Scenes of Big Data

The first technical challenge was ensuring a speedy invoicing process. The new system handles 300,000 orders in 4 hours, but this is nowhere near its maximum capacity. Our latest tests show a processing capacity of 20 invoices per second. In the future, this number can be expected to be be raised by an order of magnitude, simply by adding more instances with more consumers on them.

Another part of the work undertaken by Scandiweb was ensuring a fully automated billing flow. Communication between the payment processor (Stripe) and the Magento platform can handle a limited number of requests at a time. For this reason, an intermediary step utilizing Amazon Simple Queue Service (SQS) was introduced. The idea behind SQS is feeding the invoice tasks to a queue.

From there, we create a number of “consumers”, which are essentially independent Magento processes. Each of these consumers takes an invoicing task from the queue and performs it independently. New consumers are created and invoicing is carried out for as long as there are tasks in the queue.

If any of the invoicing requests is unsuccessful, the task is returned to the queue and tried again by another consumer. In the old system, if any of the invoice requests received an error, the whole process was interrupted and had to be started anew.

To ensure full scalability that would work with a virtually limitless amount of surveys and a wide variety of subscription box types for different market groups, two major solutions were implemented:

1. Auto-scaling

When the billing flow faces too many invoices, more server instances are created for data processing. On these instances, more consumers are automatically created in a matter of minutes, thus scaling up to accommodate any number of invoices.

Auto-scaling also works on the front end, meaning that when the company experiences more traffic than it can handle, new instances for the front-end are created. These are then added to load-balancer to accommodate for unusually large traffic.

2. Processing data via Stripe API

This is a huge portion of why the new solution is now completely scalable. Processing large amounts of data via Stripe webhooks is problematic, because while processing a large number of events, requests and responses may be received out of order and that can’t be handled by Magento 2. For instance, if the “invoice.payment_successful” event is reported before “invoice.created”, the billing process will not be successful.

In order to circumvent this, the invoice processing got rid of Stripe webhooks altogether. All data is processed via the Stripe API. This way we can control the order of events and have all invoices created in a step-by-step process.

As you can see, the new system uses Amazon SQS to carry out simultaneous invoicing tasks. With the number of consumers scalable to virtually any number, the billing flow can handle large amounts of data much more efficiently.

In the future, it’s possible that database sharding will be implemented. As a large number of consumers can potentially overload the database and disrupt the day-to-day eCommerce on the site, automated database sharding might be necessary. This means that if the load on the database becomes too great, it will be divided up and ran on multiple instances.

Things to Keep in Mind When Developing Custom eCommerce Solutions

1. First, it’s that “big data” starts small. When you can seamlessly process data, in our case invoices, measured in hundreds of thousands, scalability is the next step. If you can ensure no disruption in the quality of processing even when the number is brought up by an order of magnitude, the solution is considered fully scalable.
2. Secondly, subscription box business is booming, and its advantages are double-edged. You offer more value to the end customer than through traditional retail, and you also get a more targeted audience for paid promotion to major brands. At the same time, subscription boxes are a low hanging fruit and everyone and their grandmother is trying it out. The biggest piece of the pie goes to the one with the best technical solution and the highest margin.
3. It’s also worth pointing out that the more precisely you divide up your target audience, the more customer satisfaction will go up. For example, if a major beauty brand created a product for a very specific audience, survey results like the ones gathered by our system would provide a chance to deliver the product to a precisely pinpointed audience.
4. And lastly, there’s a thin line between getting all the info you need and being obtrusive to your potential customer. This is particularly important when dealing with mobile traffic (which is the majority of subscription box site traffic). Client grouping and profiling needs to happen seamlessly, without severely disrupting their browsing experience. In this sense, simple surveys comprised of just a few steps work wonderfully.

Do you think you’ve grown to the point of implementing big data solutions? Or maybe you’re looking for a different eCommerce solution? Scandiweb can help! Check out our service page or drop us a line at [email protected].

Related articles:

• Technology solutions for eCommerce: Bringing Virtual Reality into Everyday Shopping

Share on: