Kevin Coleman

Technical Project Manager and Developer
Home
Contact Me
Blog
May 14

wget and robots.txt
While trying to debug issues with a web application, I noticed that one of the issues was caused by web spiders crawling the site. I added a robots.txt file to the site that blocked all user agents. This would only block legitimate spiders, but at least it would reduce the number of times the spider error would show up.
 
The side effect of doing that was that it also blocked access by wget. The site I was working on also needed to support clients that retrieved software updates via wget. By default, wget honors the robots.txt files, so all updates were being blocked by my change. This behavior is is good in most cases because wget can easily be scripted to pull everything down from a site and can severely impact the performance of a site.
 
There are several ways to permit wget and block other user agents. First, if you are in control of the client, you can use a switch in wget to turn off the check for robots.txt. (See the wget docs for more info.). You can also modify your robots.txt file to permit the wget user agent or you can block all the other agents you don't want.
 
Keep the wget issue in mind as you develop distributed clients that retrieve information; It's possible that other web clients have this issue too.


12:09 PM GMT  |  Read comments(0)

March 28

Kona: A.K.A. MVC Storefront 2.0

I’ve been following Rob Conery’s MVC Storefront screencast series since September 2008. At the time, I had just discovered the MVC framework and was looking for information and tutorials on the new (for Microsoft) web application pattern. Boy, did I hit the jackpot. I got more than just information about how to build MVC apps. Rob gave viewers a solid approximation of the real-world development process using the MVC pattern and other up-to-date design patterns and development philosophies. He wasn’t afraid to make mistakes in public, critique canonical process definitions, and then change his mind and pursue other directions if things didn’t work out. He made all his decisions in public for all to see.

Now he’s on a new tack with Kona. Calling Kona MVC Storefront 2.0 is my tongue-in-cheek way to describe Rob’s desire to take the MVC storefront to the next level and inject it with a dose of Web 2.0. He wants to make Kona simple to modify, more interactive and user centric, and less dependent on any one technology stack. Yes, Rob really works for Microsoft. If this is the new face of Microsoft, bring it on.

In his latest screencast and in his MIX09 talk, he states boldly that application architecture doesn’t matter. This may sound like flame bait, and it may be, but that is not exactly his point. Architecture is just as important as it’s always been. What he is saying is that having a good application architecture isn’t enough. What also matters is the experience for administrators and end users. Just because an application my have beautiful architecture, scales well, is easily extended, and provides for all required functionality, doesn’t make an application successful in the market or in the eyes of users.

All in all, Rob is reminding developers and application builders that what matters most is what users want and need. Sound familiar?



10:22 PM GMT  |  Read comments(0)