The Web is increasingly the first go-to source for any kind of information that is needed. Unfortunately, it is often the case that while the information is free and readily accessible, it is only available in small pieces, one page at a time.

For example, you may want to know the age distribution of head of household in each zip code within a particular utility service territory. There are many zip codes and you have to call them up one by one and then cut-and-paste the information into a larger spreadsheet.

If you every find yourself in this time-consuming, mind-numbing and error-prone situation for collecting data, you may want to think about hiring Daniel to create a Web scraper for you.

A Web scraper is like having an invisible person at the computer. A customized Web scraper is built and loaded with the parameters for all of the information you need (in our example, this would be the list of zip codes that you want). The program then calls up each individual screen on the Web and captures the data screen by screen and loads it into a spreadsheet. You can let it run overnight, and in the morning you will have a spreadsheet full of all the data you were looking for. Then you can spend your time analyzing instead of collecting.

Most Web scrapers can usually be built in a day or less, costing only several hundred dollars. Daniel looks forward to each new challenge, so keep him in mind the next time you are spending too much valuable time looking up repetitive data on the Web. Individual Web scraper projects that Daniel has worked on are listed on this page. Click on the project name to get more details on what they did.

NAICS Data by Zip Code. Daniel created a VBA webscraper to pull down NAICS data from the U.S. Census Bureau Business Patterns website by zip code so the client would not have to do it manually. SAS was used to merge the Census data with California Climate Zones. The whole project took approximately five hours, start to finish, costing only $400. (2012)

Energy Efficient Product Data. Daniel created a VBA webscraper to pull down all lighting and motor data from Grainger, Service Lighting, Marathon and Baldor websites. (2007-2009)

Energy Efficiency Program Data. Daniel created a VBA webscraper to pull down all known energy efficiency and renewable energy program descriptions from the Database of State Incentives (DSIRE) on-line database. (2007)

Yellow Pages Business Types. Daniel created a VBA webscraper to pull down business types from the local Yellow Pages website and merge with customer data, based on phone number matches for MidAmerican Commercial and Industrial customers. (2007)