Why the Tasmanian filtering trial is a failure
July 29, 2008 – 8:42 pmSenator Conroy’s office could spin a thread of gold from a lump of crap, I’ll give them that. But if you’ve read the Tasmanian filtering report, it probably didn’t take long before you noticed red flags. I decided to compare the report to what Mr Conroy says in his press release and show why the trial is nothing but a miserable failure.
What Senator Conroy says: The performance or ‘network degradation’ for one of the tested products was less than 2%, whilst three products were less than 30% and two products were in excess of 75%.
What the report really says: The filter that showed less than 2% network degradation was also one of the least accurate filters at identifying illegal and inappropriate sites. The more accurate filters showed a larger drop in network performance.
What Senator Conroy says: Successful blocking (the proportion of illegal and inappropriate content that should have been blocked that was successfully blocked) was between 88% and 97% with most achieving over 92%.
What the report really says: It probably won’t take any more than 12 or 13 clicks before a filtered user can access a site containing adult or inappropriate content.
What Senator Conroy says: Overblocking (the proportion of content that was blocked that should not have been blocked) was between 1% and 6%, with most falling under 3%.
What the report really says: Even if you choose the best result (1%), out of every one million websites, 10,000 will be blocked when they shouldn’t be.
What Senator Conroy says: All filter products tested were able to block traffic entirely across a wide range of non-web protocols such as instant messaging and peer-to-peer protocols. However, most filters are not presently able to identify illegal content and content that may be regarded as inappropriate that is carried via the majority of non-web protocols.
What the report really says: The only way the filters could block traffic on non web protocols was to ban access to them completely. That means if you want to chat to Gran about her garden or drinking habit on Messenger, you wouldn’t be able to. No matter how innocent.
Those are just the starters. Here’s the main course:
1) The number of simulated users was too low. Large ISPs have hundreds of thousands of customers and even small ISPs have thousands.
2) During the trial, only 3930 URLs were filtered. When you consider Mr Conroy wants to block ‘inappropriate content’ to children, 3930 URLs is simply too low and doesn’t show the potential real impact on network performance or filtering effectiveness. The Internet contains hundreds of thousands of websites not appropriate for children by our classification standards.
3) The report claims all but one of the six filters was able to ‘filter’ HTTPS traffic. That means the Government would intercept your private and secure banking details (or PayPal transactions) and leave the door open to fraud and identity theft. Nobody wants their online purchases monitored by the Government let alone their banking put at risk.
4) There is no analysis of circumvention methods and that’s crucial to understanding why filters - ISP and software based - are ineffective. Filtering can be bypassed in minutes by a savvy net user and in hour by anyone following instructions.
5) There is no analysis of the costs of deploying and implementing a filter at ISP level, nor is there any analysis of the associated costs that will be passed onto customers.
That to me looks like a failure dressed up by a Senator with an obsession on controlling what you can and can’t see.
Visit NoCleanFeed.com to find out how you can take action. I for one wouldn’t want my blog or businesses website blocked for no reason.
Oh! If you spotted other doozies in the report, please post them in the comments. No doubt I’ve missed many.
Update: More flaws surface.
38 Responses to “Why the Tasmanian filtering trial is a failure”
Get this into tomorrow’s Crikey mate! That’s read by all the policy wonks and people who can make hay with it.
By Simon Rumble on Jul 29, 2008
Didn’t think she was quite up to Crikey standard. Maybe an extended reader comment
By Mike on Jul 29, 2008
Michael,
I agree with Simon, at the very least a letter to the editor, and to the Opposition who would love to taste Conroy’s blood.
By Terrence Valter on Jul 30, 2008
Hey, thanks for the summary. That’s one heck of a long report, and the further I get into it, the more exasperated I get.
This reads more like a promotional brochure than an actual report. It also delves into throttling P2P & non-web content toward the end, which I find a little bit trashy and presumptuous. Overall, it looks like this is going to hit the general web user more than it will hinder the curious thirteen year old.
By Ash on Jul 30, 2008
Ash, your right. It does read like a promotional brochure. It’s hard to find places where the report is critical of any of its findings.
By Mike on Jul 30, 2008
I am a network engineer and I’m reading the report. It is riddled with technical errors.
Page 9 claims that ISP filters can’t be circumvented, which is nonsense — all it takes is a tunnel to outside the filter. Exactly the same technical people use to avoid the Great Firewall of China.
Page 24 claims that subscriber bandwidth is no more than a “few megabits per second”. Which is bogus, as it is the peak customer bandwidth, not the average, which will stress the filter. That is significantly more bandwidth as 1Gbps ethernet is the cheapest way to light dark fiber.
Page 25 talks about congestion in the “prime” network. This is jargon I’ve never encountered — and I’ve been working on data networks since 1985. I’d say it has been invented for this report. Anyway, the thought that “congestion” in the network core is less worse than congestion in the access layer is totally arse-about.
Page 25 says that networks are rated on both “bandwidth” and “throughput”. Which is an odd thought since these are essentially measuring the same thing, as T=1/B.
Page 26 is totally bogus. It uses as an example a gigabit ethernet link. It then presupposes retransmissions (presumably collisions) when GbE is only available on ethernet switches, which are collision-free. Errors compound as the author then states that routers change packet sizes in response to congestion. No they don’t — routers queue packets in response to congestion and discard packets once the queue length is too long.
Figure 5 is totally bogus, as its it’s explanation: “The performance of Ethernet traffic degrades as traffic becomes increasingly ‘bursty’; that is, where packet size becomes random.”. This simply isn’t so: packet sizes remain the same as the transmitter sent them.
What’s worst about all this analysis is that it is looking at the wrong thing. It is the performance of the TCP protocol, not of the Ethernet protocol, which will dominate the user’s experience when encountering a bottleneck like a filter.
TCP performance is determined by latency, jitter and loss. Nowhere does the report measure the effect of the insertion of a filter middlebox on those essential variables. In short, these amateurs have measured the things that don’t matter and not measured the things which do.
The very worst error is in the testbed design. 80-90% of Australia’s Internet traffic goes to the USA, a considerable distance and a large amount of latency. This is not simulated in the network design — rather the testbed assumes that all the content is local to Australia and thus of low latency. Effects like jitter and loss become exponentially worse as latency increases, so the testbed understates by orders of magnitude the performance effects on traffic. This single error is so huge that its effect swamps the results, rendering the results of no value at all.
Unbelievable that my taxes have paid for such an amateurish effort.
By Glen Turner on Jul 30, 2008
And Figures 13 and 14 are a classic. Inserting some filters into the network actually improves response time. The text doesn’t note that, nor ask why.
If it is experimental error then that seems to be about 5%, and so a lot of claims about the results are invalidated by the noise of this error.
The baseline itself is worrying, it is 425Mbps. The network is a GbE switch — so where is the bottleneck? It must be the web server. That is, the tests are measuring the performance of the web server. That’s a problem because the Internet may well have web servers which can serve at 1Gbps
And there’s another glaring error in the testbed. ISP networks are designed to be resilient — to continue on even if the ISP loses a POP. The testbed shows no resiliency techniques — such as redundant filters — at all. This would considerably alter the results.
By Glen Turner on Jul 30, 2008
I would dearly love to see this on Crikey and other media sites. It’s very well written and concise, it got the point across without sending me crosseyed with jargon
By Ms Naughty on Jul 30, 2008
The reason I suggest you submit this to Crikey is it’s clearer than Stilgherrian’s pithy comment on the same report. Your length is about right too.
By Simon Rumble on Jul 31, 2008
I spoke with Crikey yesterday and they requested a 150 word version of this… no doubt for comments. Hopefully it is included in today’s issue.
@Glen Turner: Thank you very much for the more in depth tech summary! I’ll try and publish a post tonight based around your comments.
By Mike on Jul 31, 2008
Update: Stil has a China focused filtering article in today’s Crikey. My comment re Aus filtering is in the comments section.
By Mike on Jul 31, 2008
Thanks for such a detailed analysis, Glen!
Please don’t be surprised by the amateurish nature of the study. I’ve worked in govt IT before and seen first-hand what kind of moronic things go on. The general level of wastage and under-the-table deals to mates would boil most professional IT people’s blood
By est on Aug 1, 2008
Would it be possible to DDoS the living shit out of these internet filtering boxes? This is completly bullshit. Fucking nazi fucks.
By J on Oct 2, 2008
This is ridiculous…. we are becoming the next China. Maybe its time to leave the country…?
By Troy on Oct 3, 2008
On network efficiency due to “retransmits”. Just checked our core switches, between the few gig interfaces I checked (of the few hundred) I couldn’t find a single dropped frame since last time the counters were dropped.
That’s only over the last 255734561704 bytes or so per interface.
Dreadfully painful to read. It’s like the network version of reading an Intelligent Design proponent critique evolution.
By Al on Oct 14, 2008
The frustrating thing is Conroy is pushing ahead with this no matter what, and Kevin-747 is too busy buying votes to care. How can this thing be stopped? Will the Liberals even stop it?
By Billythechimp on Oct 24, 2008
I’m Strongly against the possible ISP-level filtering. I know my views aren’t important to you but seriously when did it become necessary to restrict what people can see. Have we not experienced enough that we could all see this is a bad idea. I feel freedom is what Australia is about … If we introduce this and make it compulsory it is like falling backwards… Twice!.
If in the future this happens it could lead to the government controlling what we see. I’m not just talking about porn I’m talking about site’s that have opinions like this one will most probably be extinct. Not to mention sites that are detrimental to the government that show the flaws in own society.
We need to keep our opinions and our ability to think intact. We don’t want to conform and just follow the norm… because our ability to have our own personality is what separates us from most animals and I think its not how we should live our lives.
By Ronald on Oct 30, 2008
Yeah I agree with Ronald, trying to block porn or anything people believe is evil/illegal is just stupid,not to mention very hard to block the bad websites and allow the good.It makes the governments small efforts for faster and australia wide access to the internet a waste of time. I wouldnt like to go online just to look at the playschool website lol.It is a rubbish idea and if it gets through i wouldnt be surprised if someone creates a DOS attack software and tells everyone to mass protest.The lack of faith the government has that we can choose our own websites to look at and that we can raise our own children is pathetic.
By David on Nov 12, 2008