Simple Python Web Scraper

I needed a simple html only scraper. (This doesn't use js, won't pull down data via AJAX). I found an example on another site,, but it wasn't exactly what I needed. It only pulled the data and printed it to screen. I added a list to loop through and auto saving by url name to a html file.

import mechanize \#pip install mechanize

br = mechanize.Browser()  
br.addheaders = \[("User-agent","Mozilla/5.0 (X11; U;
Linux i686; en-US; rv: Gecko/20101206 Ubuntu/10.10 (maverick)

sign\_in ="") \#the
login url\_form(nr = 0) \#accessing form by their index. Since we have
only one form in this example, nr =0.  
\\_form(name = "form name") Alternatively you may
use this instead of the above line if your form has name attribute

br\["email"\] = "email or username" \#the key
"username" is the variable that takes the username/email value

br\["password"\] = "password" \#the key
"password" is the variable that takes the password value

logged\_in = br.submit() \#submitting the login credentials

logincheck = logged\ \#reading the page body that is
redirected after successful login

urls =

for url in urls:  
req =  
filename = url.split('/')\[-1\] + ".html"  
f = open(filename, 'w')  

Which produces 2 files: