Here I am in Italy on my honeymoon, and while taking a brief pass through my email looking for something important, I see an email from Netflix about the separation of their businesses into a physical DVD division (Quickster) and a streaming video division (Netflix).
From a business perspective, I understand this completely. They want to isolate that business from the streaming business so that they can optimize the businesses separately. It will help ensure profitability on both halves (incidentally, that's also part of the reason they split up billing a month or two ago, it was a telegraphing of their moves). It may also help them limit and isolate each business from licensing terms so that they can only affect one and not both.
But from the perspective of the consumer? Well, as a consumer who didn't much care that I was paying an extra $5/month for the service (I definitely get my money's worth of 2 dvds out with unlimited streaming even at $20/month), not having the sites integrated actually pisses me off. When browsing through videos, I would usually check whether the streaming version has subtitles (my wife speaks English as a 2nd language, and finds subtitles easier to follow than spoken dialog in movies, especially British films), and if not, request the DVD. Or, when streaming wasn't available, directly request the DVD.
My interaction with Netflix is defined by video availability and feaures in the two formats in which I consume content. To separate them into two different sites makes it *more difficult* for me to choose media in a reasonable way. Three years ago, I wished for a web site that knew about my memberships to all the different sites I wanted to consume from, knew what I watched, what I wanted to watch, and told me where I could get it. Netflix is going in the opposite direction.
Remember back when they tried to remove profiles (that little feature where different people in a family could have different viewing preferences, recommendations, and number of DVDs they could have out)? You know, a practically useful feature? They ultimately kept it because so many people spoke out about it. I don't know if enough people are going to recognize how awful this recent move is and complain, but it would sure be nice to be able to know whether a movie is available in DVD and streaming without visiting two sites and jumping between them.
I've heard the stories about Netflix, about how awesome their benefits, pay, etc. are. Heck, I've even received recruiting emails from them. But you know what they need? They need to hire people who tell the upper management "no". One of the greatest bits of engineer and business productivity that we've cultivated at Adly is the ability for people to say no. We have management with great vision and ideas, a founder who can get us new designs overnight, and an engineering team that can do it almost as fast. But when you move too fast, when you are only concerned about the *right now*, you lose sight of where you are going. By saying no to daily updates, we focused on longer-term goals and company progress. Netflix needs people to say no, not because they are moving too fast (as was our problem), but because they are making decisions that make their customers angry. They need an actual customer advocate working for Netflix. Someone who talks with real customers.
I know, customers don't like change. People don't like change. We saw it continually at YouTube. But there are some people, who you have to contact and cultivate, who understand change is necessary, and who will tell you that your changes are good when they are good, and suck when they suck.
Netflix, I'm willing to be that guy. I'll do it for free: this multi-site thing sucks. Stop it.
Tuesday, September 20, 2011
Thursday, September 15, 2011
Improving Performance by 1000x
Whenever possible at Adly, we like to use existing software to solve as many problems as we can. From PostgreSQL for our database, to syslog-ng and Flask for our logging, to Redis for search, caching, and other fast-data operations, ActiveMQ for our message broker, etc. But there are times when existing software just isn't fast enough, takes too much memory, or just isn't suited for the task. This is one of those times.
In our Adly Analytics product, we calculate a number for pairs of Twitter users called "Also Follows". For two twitter users, it is the number of followers that follow both of those users. We had tried using Redis for this directly, but we were looking at roughly 700 megs of memory to store one set of 7.5 million followers. Ugh! Straight Python did a bit better at 500 megs, but that was still far more than we could reasonably use to store the data. Combine that with Redis intersections taking multiple seconds to run, and the fact that we needed to intersect 1,000 sets against 10,000 sets every week (these were due to the nature of the kinds of updates we were performing), and we'd need to have 33 processors plus roughly 300 gigs of memory to just perform this operation fast enough. That's just too expensive (over $6k/month just for machines in EC2 for the memory).
When faced with seemingly insurmountable costs, we did what any tech-directed company would do; we wrote custom software to solve our problem. What did we do?
First, we got rid of hash tables. They are great for random lookups, but in the case for fixed-size data (we had 32 bit integers for Twitter user ids), a sorted array would be roughly 1/20th the size of the set in Redis, while offering a method for random lookup (if necessary).
When we are fetching the follower lists, we get them in chunks of 5000 at a time, unsorted. We write them to temporary files on disk while we are reading them, then when done, we read the list into memory, then sort it using the C standard library quicksort. After sorting, we scan the sequence for any duplicate values and discard them. Once we have a sorted sequence of unique integers, we write them to disk.
Once they are on disk, we signal our intersection daemon to memory map the file*, letting the operating system cache the file as necessary. When we need to perform the "also follows" calculation, we run a custom intersection algorithm that is tuned to rely on memory bandwidth, which allows us to intersect a set of 5 million and 7.5 million integers with roughly 2 million shared integers in roughly 6 milliseconds on a 3ghz Core 2 Duo (using one processor). That is roughly 2 billion integers examined every second. Compare that to Redis taking roughly 7 seconds to perform that same operation (due to the random hash table lookups, allocating and copying data, etc.), and we've improved our performance by 1000x.
Our results:
For the data we need to store in memory, we went from over 300 gigs to under 15 gigs. We also went from taking around 45 minutes to intersect 1 item against 10,000, to taking 3 seconds**. While we still believe in using existing software as much as possible, sometimes it is just worth it to write what you need to in C/C++ to get the speed you need.
Along the way, we tried to use scipy.weave.inline to inline C into Python. Sadly, due to some threading issues, we got some nasty behavior and segfaults. I didn't dig too far into it and just wrote what we needed as a plain Python C extension. It's pretty easy.
* Memory mapped files are an interesting and crazy thing, available in Posix (Linux, BSD, OS X, etc.), Windows, and just about any other mature system out there. They allow you to open a file in such a way that you can access the contents of the file as if it were a C array. You can map the entire file or parts of the file. The operating system caches the data as necessary, and if you have multiple processes memory mapping the same file, the memory is shared between them.
** Theoretically it was dropped to 3 seconds, but due to processor caching behavior, Python overhead, etc., this is actually around 45-50 seconds. In an hour, we sustain roughly 2.4 trillion integers examined, or roughly 670 million integers intersected against each other every second on average.
If you want to see more posts like this, you can buy my book, Redis in Action from Manning Publications today!
Update: You can read why we didn't use a bloom filter in a followup post.
In our Adly Analytics product, we calculate a number for pairs of Twitter users called "Also Follows". For two twitter users, it is the number of followers that follow both of those users. We had tried using Redis for this directly, but we were looking at roughly 700 megs of memory to store one set of 7.5 million followers. Ugh! Straight Python did a bit better at 500 megs, but that was still far more than we could reasonably use to store the data. Combine that with Redis intersections taking multiple seconds to run, and the fact that we needed to intersect 1,000 sets against 10,000 sets every week (these were due to the nature of the kinds of updates we were performing), and we'd need to have 33 processors plus roughly 300 gigs of memory to just perform this operation fast enough. That's just too expensive (over $6k/month just for machines in EC2 for the memory).
When faced with seemingly insurmountable costs, we did what any tech-directed company would do; we wrote custom software to solve our problem. What did we do?
First, we got rid of hash tables. They are great for random lookups, but in the case for fixed-size data (we had 32 bit integers for Twitter user ids), a sorted array would be roughly 1/20th the size of the set in Redis, while offering a method for random lookup (if necessary).
When we are fetching the follower lists, we get them in chunks of 5000 at a time, unsorted. We write them to temporary files on disk while we are reading them, then when done, we read the list into memory, then sort it using the C standard library quicksort. After sorting, we scan the sequence for any duplicate values and discard them. Once we have a sorted sequence of unique integers, we write them to disk.
Once they are on disk, we signal our intersection daemon to memory map the file*, letting the operating system cache the file as necessary. When we need to perform the "also follows" calculation, we run a custom intersection algorithm that is tuned to rely on memory bandwidth, which allows us to intersect a set of 5 million and 7.5 million integers with roughly 2 million shared integers in roughly 6 milliseconds on a 3ghz Core 2 Duo (using one processor). That is roughly 2 billion integers examined every second. Compare that to Redis taking roughly 7 seconds to perform that same operation (due to the random hash table lookups, allocating and copying data, etc.), and we've improved our performance by 1000x.
Our results:
For the data we need to store in memory, we went from over 300 gigs to under 15 gigs. We also went from taking around 45 minutes to intersect 1 item against 10,000, to taking 3 seconds**. While we still believe in using existing software as much as possible, sometimes it is just worth it to write what you need to in C/C++ to get the speed you need.
Along the way, we tried to use scipy.weave.inline to inline C into Python. Sadly, due to some threading issues, we got some nasty behavior and segfaults. I didn't dig too far into it and just wrote what we needed as a plain Python C extension. It's pretty easy.
* Memory mapped files are an interesting and crazy thing, available in Posix (Linux, BSD, OS X, etc.), Windows, and just about any other mature system out there. They allow you to open a file in such a way that you can access the contents of the file as if it were a C array. You can map the entire file or parts of the file. The operating system caches the data as necessary, and if you have multiple processes memory mapping the same file, the memory is shared between them.
** Theoretically it was dropped to 3 seconds, but due to processor caching behavior, Python overhead, etc., this is actually around 45-50 seconds. In an hour, we sustain roughly 2.4 trillion integers examined, or roughly 670 million integers intersected against each other every second on average.
If you want to see more posts like this, you can buy my book, Redis in Action from Manning Publications today!
Update: You can read why we didn't use a bloom filter in a followup post.
Friday, September 2, 2011
Building for Uptime, Expecting Failure talk now available
Way back in June, I posted an article on building Adly's real-time ad network. Well, on August 1, presented at the Los Angeles Dev Ops meetup, and I just now got it uploaded to YouTube. You can view Building for Uptime, Expecting Failure now.
The full slides are available via Google Docs.
The full slides are available via Google Docs.
Thursday, August 25, 2011
Followup with Time Warner
Just before lunch yesterday, I received a call from John, a representative of Time Warner Security. They had apparently found the Reddit post and my blog entry, rolled a truck to my building, and was ... less than pleased.
We had a brief but pleasant conversation in which we both agreed that if my cable were to be disconnected again, I should call him directly so that he could have the matter resolved in a more reasonable manner. He said that they had found my note, verified that in fact I was a paying customer, replaced the red tag with a green one, and that my line shouldn't be disconnected again.
Overall, while I was dissatisfied with the offered repair time when the outage originally occurred, I was pleasantly surprised by their response at my less than elegant solution. That said, I would not recommend doing what I did.
We had a brief but pleasant conversation in which we both agreed that if my cable were to be disconnected again, I should call him directly so that he could have the matter resolved in a more reasonable manner. He said that they had found my note, verified that in fact I was a paying customer, replaced the red tag with a green one, and that my line shouldn't be disconnected again.
Overall, while I was dissatisfied with the offered repair time when the outage originally occurred, I was pleasantly surprised by their response at my less than elegant solution. That said, I would not recommend doing what I did.
Monday, August 22, 2011
How to solve your cable/internet outage with a drill
This Sunday evening, after spending the day out and about picking up some wedding stuff, getting dinner, etc., my fiance and I came home to find that our cable and internet were out. A quick reboot of the cable modem told me that just like 2 times before, when hooking up the cable for the newly arrived neighbors, the cable guy disconnected our cable.
For some reason, the local Time Warner Cable installation guys have some bad paperwork and think that there is no paying customer in my apartment, so when they come to connect someone new, because there are only 5 jacks in a cable box for 8 apartments, they disconnect me.
In the past, this was a simple problem to remedy. I would go down to where the external box is, open it up (because no cable dude bothers to lock it), add a splitter and a splice, and I'm back up. But not this time.
This time, there was an obstacle. Seems the Time Warner Cable dude locked the cable box, forcing me to call up Time Warner Cable. The conversation went something like this...
That was probably not the most prudent thing to do, but I was tired and annoyed that we lost our internet access for the 3rd time because their technicians have bad paperwork. In the past, I've even caught a tech disconnecting me (I was surfing the net at the time), and had them plug me back in.
So, what did I do? Exactly what I said I'd do. I pulled out my drill and drilled a lock for the first time (read farther down for my method). It went pretty well. The low-quality lock body was no match for my $5 bit set and $30 Ryobi.
Upon opening the cable box, I discovered that the cable guy was being particularly malicious, and had cut the splice that my cable had been using to connect (my cable line is about 8 inches too short to connect to any of the jacks directly, so I had been using a cable splice that some other cable guy had left in there). Thankfully, there was another piece that was long enough, but there were no more available jacks, and I was fresh out of splitters.
A trip to Home Depot (2 miles from my apartment) got me two splitters (a spare for next time :P) and a bunch of stuff we need for the wedding. A quick disconnect, some screwing in, and we were up. Going back inside, cable and internet were back on. Woo.
The updated box with note:
The note reads:
I'm hoping that the note and drilled out lock is sufficient to keep me connected. Time will tell.
My method for drilling a lock:
I used a 1/4" steel twist bit and drilled in the center of where the key goes (not the center of the mechanism, see the picture farther up). When I would get far enough in for the drill to vibrate a pin into the bit, the drill would sieze up and kick (I have a keyless chuck drill, so my bit would slip). I would reverse the drill, use a bit of wire to scrape the pin out of it's hole, and continue. This lock had what looked to be 8 or 9 pins, though I didn't count as I removed them.
After I had all the pins out, and could see straight through into the box, I swapped to a 5/16" wood spade bit, jammed it in the 1/4" hole, and pulled the trigger. Anything that was left in the locking mechanism (springs, etc.) were sheared off, and the mechanism spun freely.
Once my internet was up, I had a chance to check videos of the "proper" method: drill out the pins themselves, use a screwdriver to turn the mechanism while you tap the lock to get the pins to adjust enough so the lock can turn. If you fail there, you just drill out the entire lock. On the one hand, I'm a bit sad that I didn't discover the "right" way on my own. On the other hand, putting a 1/4" hole into a lock does feel really good.
Update: an update to this saga is available here.
For some reason, the local Time Warner Cable installation guys have some bad paperwork and think that there is no paying customer in my apartment, so when they come to connect someone new, because there are only 5 jacks in a cable box for 8 apartments, they disconnect me.
In the past, this was a simple problem to remedy. I would go down to where the external box is, open it up (because no cable dude bothers to lock it), add a splitter and a splice, and I'm back up. But not this time.
This time, there was an obstacle. Seems the Time Warner Cable dude locked the cable box, forcing me to call up Time Warner Cable. The conversation went something like this...
Me: Hi, my cable and internet is out.
Customer Service: Oh, I'm sorry to hear that. Is it both your cable and internet?
Me: Yes. A new resident moved in today, and the local installer plugged them in. As has happened the last two times someone has moved in, the cable dude has disconnected my cable.
(we go back and forth about my account information)
CS: Well sir, I am very sorry to hear that. I'll go ahead and give you refund because you've had so many outages.
Me: Thank you. In the past, I've just gone down there, opened up the cable box, and reconnected the cable myself. This time, the cable guy has locked the box, putting me six inches away from where I need to be to fix the problem that he caused.
CS: We can send someone out there to get this problem resolved as soon as possible.
Me: Great, when can he be here?
CS: Well, it looks like I can get someone out there on Tuesday, how does that sound?
Me: Not sufficient. Your service guy disconnected my cable, like they have done twice before, this makes it number three.
CS: I'm sorry sir, but that's the soonest I can get him there.
Me: It will take a cable guy five minutes to show up, unlock the box, reconnect my cable, and leave. He doesn't even need to knock on my door to tell me it's done. Or, he can just unlock the box, and I'll take care of it when I get home tomorrow.
CS: Sir, everyone just needs five minutes of his time, and I can't take any of our personnel off their current calls.
Me: Well, I guess you leave me with no choice. I'll just have to drill the lock and reconnect my cable.
(a short interaction that I can't remember where the customer service dude may have suggested I not do that)
Me: You know what, you don't need to bother to send someone, I'll be taking care of this. If you can send someone tomorrow morning, I recommend you give me a call at the phone number listed on my account soon, because I'm going downstairs right now to resolve this. Thank you, and goodbye.
That was probably not the most prudent thing to do, but I was tired and annoyed that we lost our internet access for the 3rd time because their technicians have bad paperwork. In the past, I've even caught a tech disconnecting me (I was surfing the net at the time), and had them plug me back in.
So, what did I do? Exactly what I said I'd do. I pulled out my drill and drilled a lock for the first time (read farther down for my method). It went pretty well. The low-quality lock body was no match for my $5 bit set and $30 Ryobi.
Upon opening the cable box, I discovered that the cable guy was being particularly malicious, and had cut the splice that my cable had been using to connect (my cable line is about 8 inches too short to connect to any of the jacks directly, so I had been using a cable splice that some other cable guy had left in there). Thankfully, there was another piece that was long enough, but there were no more available jacks, and I was fresh out of splitters.
A trip to Home Depot (2 miles from my apartment) got me two splitters (a spare for next time :P) and a bunch of stuff we need for the wedding. A quick disconnect, some screwing in, and we were up. Going back inside, cable and internet were back on. Woo.
The updated box with note:
The note reads:
Cable guy/gal, Apartment 3 is a paying customer until at least June 2012. Please stop disconnecting me. Thank you :)
I'm hoping that the note and drilled out lock is sufficient to keep me connected. Time will tell.
My method for drilling a lock:
I used a 1/4" steel twist bit and drilled in the center of where the key goes (not the center of the mechanism, see the picture farther up). When I would get far enough in for the drill to vibrate a pin into the bit, the drill would sieze up and kick (I have a keyless chuck drill, so my bit would slip). I would reverse the drill, use a bit of wire to scrape the pin out of it's hole, and continue. This lock had what looked to be 8 or 9 pins, though I didn't count as I removed them.
After I had all the pins out, and could see straight through into the box, I swapped to a 5/16" wood spade bit, jammed it in the 1/4" hole, and pulled the trigger. Anything that was left in the locking mechanism (springs, etc.) were sheared off, and the mechanism spun freely.
Once my internet was up, I had a chance to check videos of the "proper" method: drill out the pins themselves, use a screwdriver to turn the mechanism while you tap the lock to get the pins to adjust enough so the lock can turn. If you fail there, you just drill out the entire lock. On the one hand, I'm a bit sad that I didn't discover the "right" way on my own. On the other hand, putting a 1/4" hole into a lock does feel really good.
Update: an update to this saga is available here.
Thursday, July 14, 2011
A Neat Python Hack? No, Broken Code
A few months ago I saw an anti-pattern that was offered by a much-loved member of the Python community. He offered it as a neat little hack, and if it worked, it would be awesome. Sadly, it does not work. The anti-pattern is available as a recipe on the Python Cookbook here, which I alter to make fully incorrect, later switch back to the original form to show that it also doesn't work, and explain why it can't work in the general case.
A cursory reading of the interactive Python help below would suggest that the inside the try block, no other Python code should be able to execute, thus offering a lightweight way of making race conditions impossible.
When running this, I get assertion errors showing the counter typically in the range 10050 to 10060, and showing that the check interval is still 2**31-1. That means that our new counter += (1, data.decode('zlib'))[0] line, which can be expanded to counter = counter + (1, data.decode('zlib'))[0] reads the same counter value almost 90% of the time. There are more thread swaps with the huge check interval than with a check interval that says it will check and swap at every opcode!
Obviously, the magic line is counter += (1, data.encode('zlib'))[0]. The built-in zlib libraries have an interesting feature; whenever it compresses or decompresses data, it releases the GIL. This is useful in the case of threaded programming, as it allows other threads to execute without possibly blocking for a very long time - even if the check interval has not passed (which we were trying to prevent with our sys.setcheckinterval() calls). If you are using zlib (via gzip, etc.), this can allow you to use multiple processors for compression/decompression, but it can also result in race conditions, as I've shown.
In the standard library, there are a handful of C extension modules that do similar "release the GIL to let other threads run", which include (but are not limited to) some of my favorites: bz2, select, socket, and time. Not all of the operations in those libraries will force a thread switch like zlib compression, but you also can't really rely on any module to not swap threads on you.
In Python 3.x, sys.getcheckinterval()/sys.setcheckinterval() have been deprecated in favor of sys.getswitchinterval()/sys.setswitchinterval(), the latter of which takes the desired number of seconds between switch times. Don't let this fool you, it has the exact same issues as the just discussed Python 2.3+ sys.getcheckinterval()/sys.setcheckinterval() calls.
What are some lessons we can learn here?
If you want to see more posts like this, you can buy my book, Redis in Action from Manning Publications today!
# known incorrect lightweight lock / critical section
ci = sys.getcheckinterval()
sys.setcheckinterval(0)
try:
# do stuff
finally:
sys.setcheckinterval(ci)
A cursory reading of the interactive Python help below would suggest that the inside the try block, no other Python code should be able to execute, thus offering a lightweight way of making race conditions impossible.
Help on built-in function setcheckinterval in module sys:
setcheckinterval(...)
setcheckinterval(n)
Tell the Python interpreter to check for asynchronous events every
n instructions. This also affects how often thread switches occur.
According to the basic docs, calling sys.setcheckinterval(0) might prevent the GIL from being released, and offer us a lightweight critical section. It doesn't. Running the following code...import sys
import threading
counter = 0
def foo():
global counter
ci = sys.getcheckinterval()
sys.setcheckinterval(0)
try:
for i in xrange(10000):
counter += 1
finally:
sys.setcheckinterval(ci)
threads = [threading.Thread(target=foo) for i in xrange(10)]
for i in threads:
i.setDaemon(1)
i.start()
while threads:
threads.pop().join()
assert counter == 100000, counter
I get AssertionErrors raised with values typically in the range of 30k-60k. Why? In the case of ints and other immutable objects, counter += 1 is really a shorthand form for counter = counter + 1. When threads switch, there is enough time to read the same counter value from multiple threads (40-70% of the time here). But why are there thread swaps in the first place? The full sys.setcheckinterval documentation tells the story:sys.setcheckinterval(interval)Okay. That doesn't work because our assumptions about sys.setcheckinterval() on boundary conditions were wrong. But maybe we can set the check interval to be high enough so the threads never swap?
Set the interpreter’s "check interval". This integer value determines how often the interpreter checks for periodic things such as thread switches and signal handlers. The default is 100, meaning the check is performed every 100 Python virtual instructions. Setting it to a larger value may increase performance for programs using threads. Setting it to a value <= 0 checks every virtual instruction, maximizing responsiveness as well as overhead.
import os
import sys
import threading
counter = 0
each = 10000
data = os.urandom(64).encode('zlib')
oci = sys.getcheckinterval()
def foo():
global counter
ci = sys.getcheckinterval()
# using sys.maxint fails on some 64 bit versions
sys.setcheckinterval(2**31-1)
try:
for i in xrange(each):
counter += (1, data.decode('zlib'))[0]
finally:
sys.setcheckinterval(ci)
threads = [threading.Thread(target=foo) for i in xrange(10)]
for i in threads:
i.setDaemon(1)
i.start()
while threads:
threads.pop().join()
assert counter == 10 * each and oci == sys.getcheckinterval(), \
(counter, 10*each, sys.getcheckinterval())
When running this, I get assertion errors showing the counter typically in the range 10050 to 10060, and showing that the check interval is still 2**31-1. That means that our new counter += (1, data.decode('zlib'))[0] line, which can be expanded to counter = counter + (1, data.decode('zlib'))[0] reads the same counter value almost 90% of the time. There are more thread swaps with the huge check interval than with a check interval that says it will check and swap at every opcode!
Obviously, the magic line is counter += (1, data.encode('zlib'))[0]. The built-in zlib libraries have an interesting feature; whenever it compresses or decompresses data, it releases the GIL. This is useful in the case of threaded programming, as it allows other threads to execute without possibly blocking for a very long time - even if the check interval has not passed (which we were trying to prevent with our sys.setcheckinterval() calls). If you are using zlib (via gzip, etc.), this can allow you to use multiple processors for compression/decompression, but it can also result in race conditions, as I've shown.
In the standard library, there are a handful of C extension modules that do similar "release the GIL to let other threads run", which include (but are not limited to) some of my favorites: bz2, select, socket, and time. Not all of the operations in those libraries will force a thread switch like zlib compression, but you also can't really rely on any module to not swap threads on you.
In Python 3.x, sys.getcheckinterval()/sys.setcheckinterval() have been deprecated in favor of sys.getswitchinterval()/sys.setswitchinterval(), the latter of which takes the desired number of seconds between switch times. Don't let this fool you, it has the exact same issues as the just discussed Python 2.3+ sys.getcheckinterval()/sys.setcheckinterval() calls.
What are some lessons we can learn here?
- Always check before using anyone's quick hacks.
- If you want to make your system more or less responsive, use sys.setcheckinterval().
- If you want to execute code atomically, use a real lock or semaphore available in the thread or threading modules.
If you want to see more posts like this, you can buy my book, Redis in Action from Manning Publications today!
Friday, July 8, 2011
Building Adly's Twitter Analytics
If you are a regular reader of my blog, you will know that since joining Adly, I've been a little busy... Together, we have designed and built a real-time search platform, a content and location targeted ad network, and this morning we announced the public release of our Twitter Audience Analytics Platform, Adly Analytics.
This post will discuss some of the tools and methods that we used to pull and process the data to turn it into what it is today. As discussed in our press release and on TechCrunch, we have 8 tabs (12 views) worth of information that represent some of the ways that Adly influencers can view their audience data: Top Mentions, Also Follows, Top Followers, Follower Growth, Gender, City, State, and Country. In each section below, I will describe some of the techniques we use to gather and process this information.
Basic Overview
Before I dig into each of the specific tabs, I wanted to give a brief overview of the technology we use to make everything happen. There will be more detail to come on this front, and I am in the process of writing some technical whitepapers, but for now here is the big picture.
The main technical tool at our disposal is our custom Twitter Spider. Similar in a sense to GoogleBot that crawls much of the web, our spider crawls different parts of Twitter. On the more technical side of things, our spider communicates with Twitter's servers using their API.
Each different kind of data that we fetch requires a different type of spider, and each type of data is stored in one or more different formats. The underlying technology is actually very straightforward; we use ActiveMQ as a coarse-grained task queue (one of our senior engineers, Eric Van Dewoestine, who is behind Eclim, wrote our custom Python bindings about a year ago for our Ad Network), Redis as our general high-speed data store, and a custom task processor written in Python to spider, process, and store the data back into Redis.
Let's look at a few of the tabs (you can see them all on Adly.com/Analytics):
Top Mentions
The first tab, Top Mentions, is intended as a way to allow you to discover who are the most influential people that are @mentioning or retweeting you. We pull this information direct from Twitter and filter it to only represent those most influential people who are already interacting with you.
Follower Growth
The data that is behind Follower Growth is used as part of many other parts of our system. Generally, any time we receive user information from Twitter (sometimes we get it as part of the call, like in the case for Top Mentions), we check to see if the information is for a user that we have determined to be influential (in the Adly network, is from a set of the most influential Twitter users, etc.). If it is, we update the current count for their number of followers, and place that user in a list of users whose followers we want to pull. Over time, we will fetch the full listing of followers for everyone we have determined to be influential, and combine all of these lists to find new users whose information we do not yet have. Some of these users will then be influential, thus helping us to further develop our listing of influential people, find more followers, etc.
Also Follows
Once we have the full listing of followers for any two influencers, we can calculate how many followers they share. For example, 24% of @JetBlue's followers also follow @50Cent, but only 8% of @50Cent's followers follow @JetBlue. Then again, @50Cent has over 12 times the number of followers, so his followers do tend to follow @JetBlue far more than is typical. This gives both @JetBlue and @50Cent the opportunity to discover brands and influencers that have something in common with themselves.
Top Followers
Like Also Follows, since we have the full listing of followers on hand, and because we also have the public user information for all of those followers, we can easily determine things like Justin Bieber (@justinbieber) and Ashton Kutcher (@aplusk) are @charliesheen's two biggest followers. This is useful to help discover the biggest influencers that are interested in what you say, and who you may want to start interacting with.
Gender, City, State, and Country
Like Top Followers, again we have the full listing of followers on hand, but we've pre-processed the user information to determine the gender and location of Twitter users around the world. We use this information to give both an individual "62% of Kim Kardashian's followers are women" or "6.8% of Fox News' followers are in Texas", as well as "Kim Kardashian has 1.3 times the number of women following her than expected" or "Charlie Sheen has 3.2 times the number of followers in Ireland than expected".
For instance, @SnoopDogg has one of the most diverse and globally representative followings we've ever seen on Twitter. He has fans all around the world, and in many cities and states one might not expect to find a lot of Twitter users.
Beyond interesting, this is useful Business Intelligence for Snoop and his managers. Understanding who his mega fans are, and where they are most conventrated can be helpful in planning tours, personal appearances, PR and more to really bring his Twitter following to life in a meaningful way.
Anyway, I am actually headed out for some much needed R&R, so will come back to this topic later with more technical discussion and some graphics to illustrate how we created the new service.
If you want to check it out for yourself, leave a comment below with your Twitter @name, and I will be sure to get you in the queue.
Subscribe to:
Posts (Atom)