Sriram's Blog: Program to Google the Internet

Imagine your own program querying billions of web pages!!! Believe me. Its possible. Its real. With Googles’s Web API (Beta) Service, its possible. Just over a year ago, Google began to provide access to its Web API for developers. From then, a lot of applications have been developed using it. But the developed applications can’t be used for commercial use without obtaining prior grant from Google.

The sweetest thing is : You can use any programming language ( with WebServices support ) and use this API. So you can write programs using your favourite language and use Google’s service with its API. The communication between the two end programs is performed using SOAP (Simple Object Access Protocol) which is an XML based protocol for exchanging structured information in a decentralized and distributed environment.

For to start your adventure, visit http://www.google.co.in/apis/. Download the kit, create a google account and get the license key. This license key should be used in your program to get service. You are allowed to make only 1000 queries per day.

I used Java for this and my program worked. If you have a direct internet connection, things will be working fine. But if you have a proxy with NTLM authentication, then the problem starts here. Your program wont work i.e you wont be allowed to connect to Google’s service. In most of the companies, a proxy ( mostly Windows Server ) will be there for to restrict WWW connections. You will be challenged for user credentials, if you have to surf Internet. The authentication protocol used for this is NTLM ( Windows NT Lan Manager protocol ). As browsers know about this protocol, the authentication info gets exchanged properly and you are able to browse net. But your program does not know anything about this NTLM protocol and so you will not be able to make the connection. As NTLM is a proprietary protocol, there are no official specifications for it. (But some geeks have reverse engineered it and found its working!). Therefore NTLM cant be implemented in our program. So how can we get Google’s Service?

Here comes my solution. Download some free proxy server having NTLM support and install it. Hack its config file and enter your credentials which you will use for surfing in your company. Run the proxy software in your local machine. Use this ( your local machine’s IP and port ) in your program which needs google’s service. Now run your program. It will work. What actually happened is : Your program will go and contact your local proxy first. As your local proxy has NTLM support, it does the authentication part successfully with your company’s main proxy server using the credentials that you entered in its config file. This works as

YOUR SEARCH PROGRAM -> LOCAL PROXY -> MAIN PROXY -> GOOGLE’s SERVICE.

DONE !!!

Here is the snapshot of my program in action :)