Tips for new data analysts

Having worked with credit card data in the past years, I have to say that it’s a steep learning curve. My manager gave me 6 months to a year to learn what we do in our day job and he wasn’t exaggerating. I actually needed that. Even now, more than 3 years later, I still learn something new every day. The complexity and the sheer amount of data is just staggering. Our warehouse has to store data of an account on a daily basis for as long as the account is active. And an account’s life can consist of numerous interactions from applications, special offers, purchases, fees, payments to calls, mobile logins or complaints. Everything can and should be analyzed so that we can learn actionable insights that can benefit the organization.

After a tough period when I put in the work to survive, I want to share a few things that have helped me tremendously become a better analyst and programmer. My experience is with credit card data. I don’t mean to think that what I do is more complex than others, but I believe the lessons I learned can be helpful in other industries.

Learn to connect the business and the data together

At my company, the better analysts and coders often have a better understanding of the business than others. They don’t take data on face value. They use data to answer business questions and, in turn, use their business knowledge to understand data better. For example, credit cards draw much of its revenue from finance charge. However, issuers tend to appeal to potential users with an intro offer that sets interest rate at 0% for a period of time (6 all the way up to 20 months). Without that understanding, how can one understand the revenue data before and after the grace period?

Likewise, data can help issuers answer business questions. In case you don’t know, credit card companies work with the likes of Experian, TransUnion and Equifax closely for acquisition campaigns. These credit bureaus know a lot about consumers in the U.S and each has a database that can be licensed out to issuers. Issuers go through a bureau’s database and choose a population based on a variety of attributes that they believe will be most responsive to direct mail campaigns. Through post-campaign analyses, issuers learn which attributes are more predictive than others. Hence, they can become more efficient with direct mail campaigns. Let’s say that if a campaign with a 1 million pieces worth $0.5 each yields about 5,000 accounts (0.5% net response rate), each account will cost about $100 in acquisition. By using data and improving the net response rate to 0.6%, an issuer can decrease that acquisition cost to $83. That’s a real financial benefit. That’s the power of analyzing data for actionable insights.

Look at the output

A common mistake I noticed among my coworkers is that those more prone to mistakes don’t often look at the real data output. There are two reasons for it. The first is complacency. After a decent amount of time on the job, a certain level of complacency tends to develop in each analyst. Such complacency creates a false illusion that one knows everything already, while, in fact, that is often far from the truth, particularly when complex data is involved. The second reason why less effective analysts don’t look at real data output is the false assumption that their previous work will continue to deliver forever in the future. Such an assumption ignores the fact that businesses evolve all the time and when that happens, the data evolves too, such as new partners, new products, new regulations or new acquisition channels. Assuming that an old block of code will work one or two years from now is a mistake, yet luckily it’s entirely avoidable!

Find outliers

Banking data is structured most of the time. There are business rules behind the scenes that dictate how values are set. As a result, it’s very helpful to look at outliers because it will reinforce an analyst’s understanding of not only the data, but also such business rules. Let me give you an example. A co-brand portfolio of ours mandates that a customer has to maintain a specific level of spending in a calendar year to keep their Premium status. One time, I noticed that some accounts didn’t meet the threshold, yet still managed to keep their status the following year. I asked around and learned that there was a policy which enabled customers who just missed the cut to qualify for the status only if they called our Customer Care. Without paying attention to outliers, I wouldn’t be able to learn about that policy. And how did I notice those outliers? Indeed, I looked at my data output!

Read about the industry

I always believe that our understanding of the world is a network of dots and how we are able to link them together. The more we read, the more dots we add to our personal network and the better connections we can create. Without reading, there wouldn’t be many dots to connect, to begin with. Plus, the world is full of smart people who are often smarter than us. Why not taking advantage of that? We can learn so much about the trends, best practices as well as mistakes from others and apply appropriately to our business. In fact, we just launched a new credit card and I am proud to say that came from an idea of mine, an idea that was born out of regular studying of the competition and the market. So, if you want to be a better analyst, read. Read about your competition, your customers, your industry and adjacent industries too. Dig as deep as your interest allows you. Read from the basic. For instance, if you want to know about credit card transactions, read this helpful but obscure book: The Anatomy of The Swipe! The more you get to the basics, the better you can grasp more complex concepts!

Stay curious

The overarching theme over the points I make above is curiosity. As long as you stay curious, you are automatically intrigued by the questions like: How does this work? Do I miss something? Is what people told me 100% accurate? Did they miss something themselves? These questions will drive you to dig deeper and constantly ask questions; which will eventually lead you to doing all of the above and becoming an analyst.

If you come across this entry, I hope what I share above is helpful and can prevent you from making the same mistakes that I did. Leave in the comment if you have any thoughts or tips as well!

Challenges to Data Analysts – What I learned from my job

This job is the first time I have ever worked with a large amount of data and predictive methods in order to estimate the effectiveness of future marketing efforts. Below are a few challenges that I think data analysts may face:

Documentation and Universal Truth

The data and analyses can only be helpful when accepted by everyone, or at least almost everyone. Different departments need to be on the same page on definitions and calculation methods. Finance can’t have a different understanding of profit and revenue than Marketing does.

Also, the more data there is, the more important it is to have extensive and meticulous documentation. Everybody in an organization needs to be aware what each schema means, what each table means, what each column means, what each value in the column means and how the tables are connected with one another. Without a careful and detailed documentation as well as universal knowledge, an organization will encounter a lot of waste and inefficiency in operations.

Appropriate Tools

A large amount of data requires a powerful machine to retrieve it from a data warehouse. An insufficient piece of equipment such as a 4GB in RAM in a dated computer would mean hours of lost time and inefficiency. Also, it has to be said that data warehouses should be reliable. An offline data warehouse will render data analysts almost useless.

After data is retrieved, the next time is to clean, process and present data. Tools such as Tableau are awesome, but security concerns from compliance or IT can be a hindrance in adopting the tools. Plus, applications such as Tableau are expensive. If there are only a few individuals in an organization having access to it, the usefulness will be limited and the investment will not be as fruitful as it could be.

Hence, Excel is still a popular choice among organizations for data analysis. However, when it comes to processing a large dataset or using pivot tables with a lot of slices and filters, Excel is notorious for crashing more often than you can stomach. Furthermore, Excel isn’t a great visualization tool in my opinion. Presenting data to management or leadership teams usually demands sleek visuals to aid understanding and easy preparation for such visuals to save time. Unfortunately, Excel isn’t strong in either.

Connecting data to an outsourced application

Not every application that is useful to your job is internal. Sometimes, an external application is necessary. For instance, you may want to have a predictive analytics tool that can’t be built in-house. For the tool to work, you need to feed it with real data on a regular basis as much as possible since predictions often stem from historical data. However, getting data out of an organization’s walls can be a daunting task because of compliance and security concerns. Plus, ensuring that the data sent to the external application is correct and updated regularly is a time-consuming challenge. Data needs to be verified carefully. Files should be sent manually in time and in a format requested by the vendor. Automation of a data feed is ideal, but it would involve some programming and collaboration with IT and compliance.

Working at my job so far has been an eye-opener, especially in terms of working with huge datasets. I was shadowed on the job by a high schooler a few days ago. I explained to her what my job entailed and what we do everyday. I hope through this post I did shed some light on the challenges data analysts face. There are likely more, but I think these three are popular and among the biggest.