Having a lot of data is powerful. Handling is NOT easy

Brands’ blunders in their emails

Over the past few weeks, I had a couple of incidents in which brands sent me pretty awkward emails. The first incident was with the online pet store 1800PetMeds. A few weeks ago, I found out that my cat had ringworms. I went to 1800PetMeds to buy an oral solution that his vet prescribed. The transaction took place on 21st February 2021 and went smoothly; which I was thankful for. The bottle’s capacity is 52ml. My cat is supposed to take 1ml per day for 7 consecutive days on alternate weeks, meaning that we can’t use up that bottle in 30 days. Even if he takes it every day, it will have to take around 50 days to finish the bottle. Nonetheless, 1800PetMeds waited only for 30 days before they sent me an email encouraging me to buy another bottle. Their email headline read: Kimi’s next Intrafungol order is read. While I appreciate their initiative, I prefer my cat being healed completely to having to buy another bottle.

Another incident was with Hulu. It sent me an email at 10AM in the morning with a one-month free trial as a gift for my birthday. While the note was late by some margin, I still appreciated it. Less than 12 hours later, it sent me another email at 7PM, asking me to become a subscriber. If that’s not awkward enough, here is the kicker: I already received the same trial offer a few days ago and took it! I doubt that there is a system in place at Hulu that manages the delivery of marketing emails.

Handling a lot of data isn’t easy

1800PetMeds and Hulu aren’t some mom-and-pop shops that don’t have the resources to acquire and analyze data. On the contrary, they are Internet companies that should be experts in data analytics. Yet, they still have blunders like my examples above. To be clear, there is not a human-being sitting at a desk and sending out emails like above from Outlook. They are all automated from email tools such as Mailchimp. Hence, this is a product of my information being stored in their database and their operationalizing it.

This post isn’t to ridicule them. Since I have first-hand experience in dealing with data and knowing how difficult it is, I feel for them. At work, I deal with credit card data. There are many partners in our portfolio, some of which can have hundreds of thousands of accounts. Many accounts have hundreds of thousands of transactions every year. The sheer amount of transactions, coupled with their randomness in frequency, makes it a monumentally challenging task to figure out the purchase pattern for each person so that we can offer personalized marketing. For good measure, depending on how good your payment processor and internal data system are, the problem can be compounded by the irregularities in merchant descriptions. Below is what I have to do at work to categorize purchases into merchants. Think about what it is to do it for so many merchants out there

This is just one of the many aspects of what my job entails. We also have to look at how some attributes such as FICO, Balance, Credit Limit changed over time, how an account is engaged digitally (whether it enrolls online, mobile, e-statement, billpay, auto-pay or whether it is connected to a digital wallet), how profitable an account is and where that profitability comes from, and how we can be more efficient in acquiring account (whether direct mail, Internet, our retail branches, our Financial Institution partners or our Cobrand partners’ stores are the most efficient channels).

When I first joined my current team, my boss told me that it would take me a year or at least 6 months to be comfortable, not yet proficient, with what we do on a daily basis. He wasn’t wrong. Our learning curve is very steep. Plus, when dealing with a large amount of data, you have to take into account the infrastructure elements. Here are just a few on a high level

  • Is your current data infrastructure set up to assist fast data retrieval?
  • Does your data warehouse have high availability? Or does it crash a lot?
  • Is it easy to get the data you need or does it take hours to run complicated SQL queries?
  • Is there a set of universal definitions of metrics and fields?
  • Do you have a data visualization that can aid in presenting complex data? Is it connected straight to the data warehouse? Is it in a coding language that requires your team to learn?
  • Do you have a machine learning capability in-house to create proprietary models?
  • Is there a tool that can help eliminate biases to create apple-to-apple comparisons? If yes, how is data transferred from internal data warehouses to that tool?

I don’t believe my company is elite in data analytics. Not even close. I don’t know for sure, but companies like Netflix or Google should be an exciting place to work at because you’d be able to see how they handle an ocean of data at their fingertips. For many companies such as my employer, even though data driven operations are worthy visions, they are highly difficult to realize. Well, so is making money, I guess.