Clicks or Click Fraud?
Check out animation about millions of bots clicks show at Bit.ly as "statistics"
Twitter: The Dark Side Study
- EXECUTIVE SUMMARY
- INTRO to SOCIAL NETWORKING
- Human Society as Social Medium
- Networking at the Speed of Light
- Man I Need Coffee so Bad
- WELCOME to REAL-TIME WORLD
- The First Tweet?
- Businesses Emerge
- CLICK FRAUD - THE DARK SIDE
- URL Shortener
- Bit.ly’s Vulnerabilities
- Bit.ly – Twitter’s choice
- Tricky Analytics
- Twitter's ECO Footprint
- Are We Getting Stupider?
- Twitter Frenzy
- What Kind of Future with Twitter?
- EXPERIMENTS - Bit.ly Validity
- BOTS vs. HUMANS ratio
- BOTS vs. HUMANS by AdSense
- Insight Into Followers
- Bots Folllowing Bots Following Bots
- Christians Following Porn Bots
- Celebrities - Bots of the Worst Kind
- Celebrity Poluters
- Obama Girl?
- Direct Messaging Value
- MILLION CLICKS - ZERO HUMANS
- CONCLUSION
- Appendices
- BOTS vs. HUMANS IPA Analysis 1
- Followers Breakdown
- BOTS vs. HUMANS IPA Analysis 2
Check out these Ads and Click ones of Your Interest
Twitter: The Dark Side | Bit.ly’s BOTS vs. HUMANS Clicks Ratio
Previous discovery -- that Bit.ly underreports raw, total number of clicks delivered to their short URLS -- would not be such a bad thing if the clicks Bit.ly reports were valid, having actual humans behind the click, but this is far from the truth. When we compared Bit.ly’s statistics with two other reputable analytics services, StatCounter’s and Google Analytics the differences were astonishing. All Bit.ly really tells you is click count somewhere between the true raw click count and the valid human click count though they dress it up in a fancy graphical presentation.
For Bit.ly’s total of 1,064 clicks Google Analytics has shown 34 and StatCounter 36 unique visitors. Let’s split GA & SC analytics and take the average of 35 unique visitors for 1,064 “clicks”.
That data shows that only 3.29% of all clicks registered on Bit.ly were valid, unique human clicks and the astonishing 96.71% were coming from non-human entities. In this case Bit.ly over-reports the number of clicks for 96.71%.
We then embarked on a grueling process of confirming that a click, as registered in a raw data gatered by our own URL shortner, Dac.im, belongs to either a human or a bot. Files are analyzed on a number of levels. Many user agents could be found in various on-line databases but bot detection that depends on the http USER AGENT HEADER is still a long ways away from being 100% accurate because there will always be new user agents that aren't in any of the files yet and hence might not be properly identified. Further malicious bots can easily spoof a USER AGENT HEADER, which is after all simply a string of text put into the http header by the user agent. A bot can claim to be anything as we shall show in our final experiment.
Bit.ly does not provide RAW data, just their own, generic and misleading data thus the need for an accurate source of data as gathered by Dac.im logs.
The bottom line is simply this; relying soley upon the USER AGENT HEADER for bot detection is completely unreliable. It is like asking criminals to voluntarily where a striped suit every time they go out of their house.
We dived into IPA (Internet Protocol address) and RDNS (Reverse Domain Server Name) of every click and checked each and every IPA with:
- https://www.arin.net/index.html ARIN, the regional registry of IP and ASN numbers for North America, South America, the Caribbean, and sub-Saharan Africa.
- http://www.botsvsbrowsers.com/ a large database that lists user agents in categories and distinguishes between robots and browsers,
- http://www.projecthoneypot.org/index.php that is a free, distributed, open-source project to help website administrators track, stop, and prosecute spam harvesters stealing email addresses from their sites,
- http://www.domaincrawler.com/ a website that provides d omain information, whois & dns reports.
We also searched for more data about IPAs that were difficult to asses as bots or humans, we googled them, we read about them on forums, we checked blacklisted IPA and compared them with difficult to assess IPAs and at the end we made a call on each IPA that showed up in our logs.
The FULL LIST of User-Agents clicking on Bit.ly and our own URL Shortener, Dac.im, and the logic behind every call is in our BOTS vs. HUMANS Analysis that can be found in our endnotes or by clicking here.
We realized that the truth on statistics Bit.ly provides is that it statistics are egregiously inaccurate at best and fraudulent at worst. Bit.ly presents users with phantom numbers that counts cyberspace’s ghosts and drones, robots and crawlers, presenting them as they are all humans, something they are not.
PART XIV: Bit.ly' s BOTS vs. HUMANS ratio by Google's AdSense
Licensees may copy, distribute, display and perform the work and make derivative works based on it only if they give the authors the credits and only for noncommercial purposes.
For on-line displays of the work and derivative works licensees must include SEO Artworks link as follows: http://www.seo-artworks.com/Twitter/twitter-study.htm.
SHARE YOUR THOUGHTS
