bawnweb.com

bwSearch's bot for the creation of a Graphical Search Engine has stopped crawling. We have downloaded over 115 thousand pages and have a database of 7 million links to sort through at this time.

bwSearch .1 is crawling the web now. So far I have noticed some bugs:

Hyperlinks for protocols other than http get mangled and added to the end of an http:// link. this causes the bot to request nonexistant pages with mailto: links quite a bit. -- May 16th, 2008

Files other than web pages are being downloaded and parsed -- May 17th, 2008

My bot is crawling the internet downloading pages, then following the links on those pages to get to more pages. In the process, the link text is being saved so that I can create a 'tag cloud' of the internet, using the links people have made.

If you experience an issue with bwBot visiting your site, or disobeying your robots.txt (it shouldn't) please contact me pabawn [at] gmail [dot] com. Please provide snippet of your web server log so I can improve my bot.

Known Issues

I am creating my own search engine in order to learn PHP better.

The search engine will be seperated into a variety of modules. A crawler will need to visit websites of interest, then upload information about sites that it visits to a MySQL database. The crawler will also record the links that it discovers so subsequent instances of the crawler have "new" sites to visit.

A seperate program called the indexer must read information from the MySQL server and decide what to do with it. It will create an 'index' of sites that the crawler has visited.

I have more questions than answers when it comes to deciding if the GDP is an indicator of quality of life. I believe this is a good thing because it shows I am learning and exploring. Gross domestic product is the sum of all investments of capital, net exports, government spending, and consumption. The GDP does not not count labor that does not get paid for (think time spent cooking and cleaning one's own home). The GDP is oft used to measure the health of our economy; indirectly measuring the quality of our lives.

One argument for returning to local food production is that the environmental cost of transporting food over long distances is too high. Average food in the United States is shipped 1500 miles from where it was grown or produced to the consumer, according to McKibben's book, Deep Economy. Those in support of local food production argue that food travels too long distances and thus is a major contributor to carbon emissions. McKibben writes that our food “arrives at the table marinated in [crude oil]”.

Arguments against a paid vacation law in the United States could be made by more than one entity (unfortunately). Here are some arguments that have been made:
- Employees do not want to vacation due to the possibility of work piling up in their absence. Of workers who are given paid vacation opportunities in their companies, some already refuse vacations in order to avoid this work piling up.
- Employees lack the cross-training to cover for absent vacationing coworkers, so legally required paid vacation could harm United States businesses.

Economists need to use whole-systems thinking to determine the real price of doing business and how successful an economy will be at being sustainable in the long run. Biology classes like Design in Nature at Evergreen have taught me about natural capital and the biological systems at works within it. What's the Economy for anyway has shown me that accountants and traditional economy studies do not “take into account” biological systems, however when I took microeconomics at South Puget Sound Community college we did talk about externalities and finite natural capital.

After upgrading my system from Ubuntu linux 6.06 to 8.04 I noticed that Firefox refused to install google toolbar from toolbar.google.com. I imagine this might be a problem for windows users too, but I'm staying away from beta software on my aging windows machine.

On Ubuntu 8.04, Firefox said google toolbar had failed a compatibility check even though the green 'installing' bar filled to 100% after I chose accept on the the installation popup screen. A quick google search showed other ubuntu users were having the same problem and people complaining about this. Here is how you solve it.