<body><script type="text/javascript"> function setAttributeOnload(object, attribute, val) { if(window.addEventListener) { window.addEventListener('load', function(){ object[attribute] = val; }, false); } else { window.attachEvent('onload', function(){ object[attribute] = val; }); } } </script> <div id="navbar-iframe-container"></div> <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script> <script type="text/javascript"> gapi.load("gapi.iframes:gapi.iframes.style.bubble", function() { if (gapi.iframes && gapi.iframes.getContext) { gapi.iframes.getContext().openChild({ url: 'https://www.blogger.com/navbar.g?targetBlogID\758334277\46blogName\75Sriram\47s+Blog\46publishMode\75PUBLISH_MODE_BLOGSPOT\46navbarType\75BLUE\46layoutType\75CLASSIC\46searchRoot\75//metallicatony.blogspot.com/search\46blogLocale\75en\46v\0752\46homepageUrl\75http://metallicatony.blogspot.com/\46vt\0756657637821817248025', where: document.getElementById("navbar-iframe-container"), id: "navbar-iframe" }); } }); </script>

Friday, July 03, 2015

All Posts

Labels:

Wednesday, May 13, 2015

REST APIs to perform Geospatial operations using MongoDB, Google Geocoding API, GeoJSON and Spring Data

This project is an extension to the previous project “How to build CRUD APIs using Apache CXF, Spring Data and MongoDB to manage a set of employees”. The objective of this project is to extend the APIs that were built out of the previous project, to deal mainly with employees’ address information. To accomplish this objective, it utilizes and thereby demonstrates the power of MongoDB’s geospatial features, Google’s geocoding apis and GeoJSON. Some of the real world use cases like the below are something that is possible using this project

a) List all employees that live within a 50 mile radius of a given address
b) List all employees that live within a given set of polygon co-ordinates

Before proceeding further with the implementation, here is a brief introduction about the supporting geo-technologies used in this project

GeoJSON
GeoJson is an open standard format developed to represent spatial geometries using standard JSON name/value pair convention. Geometries could be a simple spatial Point, LineString or a complex Polygon. In addition that, GeoJson also supports geo-features and feature collections. A “geo-feature” is typically a geometry combined with its additional properties and a feature collection is a multitude of features bundled as a single representation.

A geometry object mostly contains two members – “type” and “coordinates”. Type represents the name of the geojson object and coordinates represents an array of longitude and latitude pairs. There can be many such co-ordinate pairs depending on the type of geometry being represented. A couple of sample GeoJSONs are shown below
{ "type": "Point", "coordinates": [100.0, 0.0] }
{ "type": "Polygon",
    "coordinates": [
      [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]
     ]
}
More about GeoJson and its examples can be found here GeoJSON Specification - http://geojson.org/geojson-spec.html#appendix-a-geometry-examples

MongoDB’s Geospatial features
MongoDB supports a handful of geospatial operations on both Euclidean plane and spherical surface. These operations are possible because it is equipped with two types of geo-indexes – 2dindex and 2dsphere. 2d index supports operations on an Euclidean plane and allows mongo documents to have geo-information as legacy co-ordinates [latitude, longitude]. 2dsphere index supports operations on earth like sphere and supports geo-data as both legacy co-ordinates and as GeoJSON.

Based on the type of operations that might require for an application, a choice needs to be made from either of two structures and that in turn will decide the type of geo-index that needs to be used. More about the different types of operators available in MongoDB is over here MongoDB GeoSpatial Query operators - http://docs.mongodb.org/manual/reference/operator/query-geospatial

Google’s geocoding Api
Geocoding is the process of converting a regular postal address into geographic co-ordinates. Reverse geocoding is the reverse process of converting geo coordinates back to human readable addresses. Using these processes, it is possible to encode addresses to coordinates and perform geo-operations on the coordinates.

More about Google’s Geocoding API can be found here Google developer site - Geocoding - https://developers.google.com/maps/documentation/geocoding/. An OAuth application (Google calls it project) has to be created and activated in Google's developer site to use this api. The server key of the created OAuth project will be used in our sample application to geocode addresses.

Having talked about the geo-technologies used in this project, let’s move on to discuss about how to incorporate these features and perform a real world usecase of "finding employees who work within a given radius of 50 miles from a given address".

Modifications to MongoDB document structure
As the employee’s work address information needs to be persisted along with the rest of employee information, the existing “employees” mongo document structure (the one that was used here “How to build CRUD APIs using Apache CXF, Spring Data and MongoDB to manage a set of employees”) is extended to comprise the "address" geojson. The existing documents in the store are updated to have their corresponding employee's address so that they qualify as candidates for the usecase that we are working towards. In addition to that, the documents that will be created in future needs to have the address information. Again, it is not mandatory that every employee’s document needs to contain their work address, but if they have, such documents will qualify as a candidate for geo operations.

The “address” object in our case is not a straight forward geojson. It contains two other objects – “postalAddress” and “location”. Postal address carries the work address of the employee. Location is the actual GeoJson object that carries the geocoded co-ordinate information of the employee work address. These two objects are bundled together into a single address object. Below is a sample address json object
"address" :{
    "loc": {
        "type": "Point",
        "coordinates": [
            -122.0829,
            37.4211
        ]
    },
    "postalAddress": {
        "street": "1600 Amphitheatre Parkway",
        "city": "Mountain View",
        "state": "California",
        "zip": "94043",
        "country": "UnitedStates"
    }
}
Following mongo query inserts documents that includes the above address structure into the employees collection of organization database
> use organization;
> db.employees.insert({
    "empId": NumberLong(35),
    "fname": "Richard",
    "lname": "Stallman",
    "deptName": "CS",
    "salary": 1000,
    "address": {
        "loc": {
            "type": "Point",
            "coordinates": [
                -122.3947492,
                37.7899519
            ]
        },
        "postalAddress": {
            "street": "199 Fremont Street",
            "city": "San Francisco",
            "state": "CA",
            "zip": "94105",
            "country": "United States"
        }
    }
});

Once we have a handful of employees along with their work address in mongo store, it is now time to query them using mongo console (or any favorite mongo client of yours) to find whether they are really query-able. MongoDB cannot support any geo-operations without having a geo-index. If done so, it says there is no index. Below is a snippet to illustrate that

Query before Indexing
> db.employees.find({
    "address.loc": {
        $near: {
            $geometry: {
                type: "Point",
                coordinates: [
                    -122.08,
                    37.42
                ]
            },
            $maxDistance: 80467.2
        }
    }
});
error: {
        "$err" : "Unable to execute query: error processing query: ns=organization.employees limit=0 skip=0\nTree: GEONEAR  
field=loc maxdist=35000 isNearSphere=0\nSort: {}\nProj: {}\n planner returned error: unable to find index for $geoNear query",
        "code" : 17007
}

Create a 2dsphere geo index
A 2dsphere index is created on the GeoJSON object. In this case, it is “address.loc” which is an embedded document.
> db.employees.createIndex({"address.loc": "2dsphere"});
{
    "createdCollectionAutomatically": false,
    "numIndexesBefore": 1,
    "numIndexesAfter": 2,
    "ok": 1
}

> db.employees.getIndexes();
[
    {
        "v": 1,
        "key": {
            "_id": 1
        },
        "name": "_id_",
        "ns": "organization.employees"
    },
    {
        "v": 1,
        "key": {
            "address.loc": "2dsphere"
        },
        "name": "address.loc_2dsphere",
        "ns": "organization.employees",
        "2dsphereIndexVersion": 2
    }
]

Query after Indexing
After creating a 2dsphere index, the same geo query operation is successful
> db.employees.find({
    "address.loc": {
        $near: {
            $geometry: {
                type: "Point",
                coordinates: [
                    -122.08,
                    37.42
                ]
            },
            $maxDistance: 80467.2
        }
    }
});

{
    "_id": ObjectId("55227f844beece3fbb066d7e"),
    "empId": NumberLong(33),
    "fname": "Sundar",
    "lname": "Pichai",
    "deptName": "CS",
    "salary": 1000,
    "address": {
        "loc": {
            "type": "Point",
            "coordinates": [
                -122.0829,
                37.4211
            ]
        },
        "postalAddress": {
            "street": "1600 Amphitheatre Parkway",
            "city": "Mountain View",
            "state": "California",
            "zip": "94043",
            "country": "United States"
        }
    }
}
The next set of steps is to make modifications to our java Crud apis to support and utilize these features

POM changes
The POM of this project is almost the same like the previous project apart from the fact that this one includes google’s geocoding library. Below is a snippet to do that
0.1.6
 
  com.google.maps
  google-maps-services
  ${google.maps.services.version}
 

Create Employee
The create employee api request is added with a new property called “Address”. This address will represent the work address of the employee. The below implementation geo-codes this given address into latitude and longitude information and then persisted into mongo store using the above document structure.
public EmployeeResponse createEmployee(EmployeeRequest employeeRequest) throws AddressException, Exception {
 EmployeeResponse employeeResponse = null;
 Employee employee = null;
 Double[] latlng = null;
 
 if (employeeRequest != null) {
  Long empId = employeeRepository.getNextId();
  log.info("new employeeId={}", empId);

  latlng = employeeHelper.getGeoResult(employeeRequest.getAddress());
  employee = employeeAdapter.buildDocument(empId, employeeRequest, latlng);
  employeeRepository.createEmployee(employee);
  if (employee.get_id() != null) {
  employeeResponse = employeeAdapter.convertToEmployeeResponse(employee);
  }
 }
 
 log.info("employeeResponse={}", employeeResponse);
 return employeeResponse;
}

The given postal address is converted to a string representation which is then used by the geocode call. The geocode call also uses something called as “geocontext”. It is the context created using the server key that was generated from your OAuth project in google’s developer site.

private static final GeoApiContext geoContext = new GeoApiContext().setApiKey("xxx");


/**
 * Gets geo information for a given postal address using google geocoding api
 * @param address the address information
 * @return GeocodingResult[] an array of GeocodingResult
 * @throws Exception
 */
public Double[] getGeoResult(PostalAddress pa) throws AddressException, Exception {
 log.info("get geoinfo for address={}", pa);

 if (pa != null) {
 String address = employeeAdapter.convertPostalAddressToTextAddress(pa);
 return geoCode(address);
 }
 return null;
}

 /**
  * Gets geo information for a given address using google geocoding api
  * @param address
  * @return Double[]
  * @throws AddressException
  * @throws Exception
  */
 public Double[] geoCode(String address) throws AddressException, Exception {
  log.info("geocode address={}", address);
  GeocodingResult[] geoResult = null;
  
  geoResult =  GeocodingApi.geocode(geoContext, address).await();
  // if no matches or more than 1 match throw address exception
  if (geoResult == null
    || geoResult.length != 1
    //|| geoResult[0].geometry.locationType != LocationType.ROOFTOP
    ) {
   throw new AddressException("INVALID_ADDRESS");
  }
  log.info("received geoInfo={}", geoResult[0].geometry.location);
  return new Double[] {geoResult[0].geometry.location.lat, geoResult[0].geometry.location.lng};
 }

Get and Get all employees
The most interesting usecase in get and get all employees apis is to find out all the employees in the organization living within a given radius (in miles). To elaborate a bit more, this operation finds all employees who live within a given radius of a circle whose center falls at the address that was provided in the request. This is accomplished by geo-coding the provided address into lat-long co-ordinates using geocoding api. Once we have the co-ordinates, they can be used to query mongodb using $near query operator in such a way that those co-ordinates are the center of a circle. The $near query operator in turn uses the geospatial index created on the "employees document collection" to find whether such an employee exists.

else if (latlng != null && lname == null) {
   // get employees that live within a given radius of the given address
   NearQuery nq = NearQuery.near(latlng[1], latlng[0], Metrics.MILES).maxDistance(new Double(radius));
   GeoResults<Employee> empGeoResults = mongoTemplate.geoNear(nq, Employee.class);
   if (empGeoResults != null) {
    empList = new ArrayList<Employee>();
    for (GeoResult<Employee> e : empGeoResults) {
     empList.add(e.getContent());
    }
   }

The below shown code snippet is to find all employees that have a given last name and live within a given radius (in miles) of the given address. The below method is different in a way where the query is completely created manually using Spring data's BasicDBObjectBuilder. The fetched DBObject is iterated using DBCursor and converted to "Employee" mongo document automatically using MongoConverter.
else if (latlng != null && lname != null) {
   // get employees that have a given last name and live within a given radius of the given address
   DBObject geoQuery = buildGeoQuery(latlng, radius);
   DBObject addressLoc = BasicDBObjectBuilder.start().add("address.loc", geoQuery).add("lname", lname).get();
   DBCursor cursor = mongoTemplate.getCollection("employees").find(addressLoc);
   empList = cursor.hasNext() ? new ArrayList<Employee>() : null;
   while (cursor.hasNext()) {
    DBObject empDBObject = (DBObject)cursor.next();
    Employee e = mongoTemplate.getConverter().read(Employee.class, empDBObject);
    empList.add(e);
   }
  }


 /**
  * Builds geo query using latitude, longitude co-ordinates and radius. Given
  * radius is in miles and needs to be converted into meters.
  * @param lname
  * @param latlng
  * @param radius
  * @return
  */
 private DBObject buildGeoQuery(Double[] latlng, String radius) {
  // Mongo uses longitude, latitude combination
  Double[] coordinates = new Double[] {latlng[1], latlng[0]};
  DBObject geometryContent = BasicDBObjectBuilder.start().add("type", "Point")
    .add("coordinates", coordinates).get();
  DBObject nearContent = BasicDBObjectBuilder.start().add("$geometry", geometryContent)
    .add("$maxDistance", (Double.valueOf(radius) * 1609.34)).get();
  DBObject addressLocContent = BasicDBObjectBuilder.start().add("$near", nearContent).get();
  return addressLocContent;
 }

The complete source code of this project has been hosted in my GitHub repository – REST APIs to perform geospatial operations using MongoDB. Please feel free to peek and play around.

Labels: ,

Monday, February 09, 2015

Tweetit - a simple bulk tweeter using twitter APIs and twitter4j

Tweetit is a pretty simple java based command-line bulk tweeter application capable of automatically tweeting, retweeting and cloning existing tweets from twitter stream. Tweetit was primarily created for the ones who work for twitter campaigns fighting for a good cause. This application can be further extended to schedule tweets using readily available scheduler tools that come pre-installed with any operating system. It may also be useful for product marketing and sales announcements using a trending hashtag to get maximum reach.

I’m thankful to twitter4j and twitter APIs without which Tweetit could not have been created. Tweetit has been open sourced under Apache 2.0 license and hosted in github. Feel free to download the source from GitHub repository - Tweetit and play around.

Origin of Tweetit
        A super frustrated group of good-hearted people lost their patience waiting for the American government to finalize and publish a rule RIN:1615-AB92, also called H4EAD – a provision to provide work authorization for certain H4 spouses who have been backlogged heavily due to the unavailability of immigrant visas (green cards). They have been waiting for this rule to effect since 2012. Out of extreme frustration, this group made a decision in trackitt discussion thread to start running campaigns through various mediums. There were some from the community that sent emails to their local congress men, there were some who wrote post cards to USCIS office in Washington DC, there were some who sent fax to DHS and USCIS, there were some who started aggressive tweeting and finally there were some who met with USCIS directors & law offices to get updates and represent the whole community. The members of this community were separated by distance but unified by their thoughts and actions.

Out of all such campaigns, twitter campaign picked up a lot of steam after USCIS director Leon Rodriguez stated that he understands their frustrations and knows about the tweets and emails that are being sent. He further recommended to keep the momentum and noise high to get things done sooner. After listening to his suggestions, this community started working more towards their campaigns in which twitter became the primary medium of expression. This created a need for a simple bulk tweeting application that will help everyone to work more towards this good cause.

Capabilities
Tweetit is capable of
  • Searching tweets under a specific hashtag from twitter stream and retweeting them back in your timeline
  • Searching tweets under a specific hashtag from twitter stream, cloning the searched tweets and tweet them to your timeline as new tweets. I call this cloning!!
  • Tweeting status messages and pictures that can be picked from a source of data, currently being a spreadsheet. The pictures need not be embedded in the spreadsheet but just an absolute path (reference) to pictures in your hard drive has to go in the spreadsheet

Being Ethical
        Twitter is a wonderful social medium through which voices of unheard and unseen can be listened to. Social media listening tools monitor and measure trends of tweets to learn and aid organizations to make informed decisions. Campaigns take place in twitter to express support and hate, likes and dislikes, agreements and disagreements to spread the cause, news and views. Therefore, it is not right to spam such a useful social medium and disrupt the system. It is indeed not ethical to do it.

Twitter officially encourages everyone to build tools and applications over the features it provides. However it throttles the traffic based on various measures so that a higher volume traffic source does not choke a lower volume traffic source. Twitter allows a maximum of 2400 updates (includes tweet and retweet) per day and it further breaks down the limits per 15 minute window - https://support.twitter.com/articles/15364-twitter-limits-api-updates-and-following. Some of the known throttling mechanisms that twitter already uses are IP based and APIs based. It has quotas per API per application. Users that are often flagged and suspicious are moved to a lower tier or sometimes even blocked permanently.

Tweetit application has been built completely being conscious about these principles. It neither tries to spam nor encourages you to do so. It has been built with just one good objective - to ease the life of twitter-heavy users that mainly run good-cause campaigns. It accomplishes this by pausing a minimum of 10 seconds in between tweets & retweets. Apparently, generous users can increase the interval between tweets but cannot bring it lower than the minimum allowed. This helps users two folds
  • To be a good twitter user paving way for rest of the world to work for their respective needs and causes
  • Do not fall under twitter’s radar-of-scrutiny and take the brunt of being labelled as a flagged-user

Tweetit Prerequisites
  • Tweetit runs in any operating system that supports Java run time environment (>= version 1.6). Tweetit is not a web application and can be just run from the command-line or shell prompt using its executable jar file.
  • Supports tweeting text and images. Currently it supports only JPEG (.jpg) images.
  • It is not required to have Microsoft Office (or any other ports of it) installed but it will be good to have, if you would like to add and edit the contents of spreadsheet. Please note that Tweetit supports (.xls) spreadsheets but currently does NOT have the support for newer (.xlsx) spreadsheets yet.
  • And lastly, if you have a basic technical ground or has a very basic exposure to java, it is pretty easy to catch up the rest of instructions and use this app

Tweetit will NOT need your Sign-in password but needs your OAuth keys to tweet on your behalf. It uses a special file called key file. This key file is expected to contain the user’s OAuth keys for twitter authentication. These keys can be generated at https://apps.twitter.com. OAuth keys are a secured way to access and post tweets on your behalf. Below steps will guide you to generate needed keys and access tokens. It's pretty simple to do and it will not take more than 15 minutes. Once created, these OAuth keys can be used forever to run tweetit. It is just an one time effort. Please note that, you will have the full control over your keys and contents of tweets. Tweetit does not and cannot misuse your keys. Neither it can snoop or mess with your tweet contents.

Generate OAuth keys
OAuth is a pretty widely used Open source authorization standard that any client application can use to get secure access to users resources without the need of their login credentials. Below is a onetime setup steps that has to be followed to generate OAuth keys from your twitter account. Once done, these keys can be used to run “tweetit”.

1) Go to https://twitter.com/settings/devices and add your phone number. Twitter validates your phone number by sending a verification code. Once done, UNTICK all the check boxes unless you want to receive twitter text notifications to your mobile. Kindly be aware that twitter does not allow you to generate OAuth keys if you do not have a phone number added to your profile. A screenshot that shows all text related settings of mobile deselected.



2) Go to https://apps.twitter.com/ and create a new application through which you can create keys. Please note that the application name should be globally unique. Click “Create your Twitter application” after accepting the Developer agreement. Below is a screen capture of the details after having entered.



3) Once your application is successfully created, navigate to “Permissions” tab/link and provide “read and write” permission as shown below. This ensures that your keys can be used by Tweetit to post tweets on your behalf.



4) Click “Keys and access tokens” tab/link to display application settings including your consumer key and consumer secret. Scroll down to “Your Access token” and click “Create my Access Token” to create access tokens.



5) Now you got all the OAuth keys and access tokens that are needed. The values you just got for a) Consumer Key b) Consumer Secret c) Access Token and d) Access Secret
need to be copied to a text file called keyfile.txt as shown below. A sample keyfile.txt can be downloaded from github repository https://github.com/metallicatony/tweetit/blob/master/src/main/resources/keyfile.txt. If you find that the contents of the downloaded keyfile are all present in the same line, it is because of End of line conversion differences between unix and windows. Kindly make the file look like the below screenshot as a first step. Then replace the respective values of keys and tokens generated from the previous step, in place of dummy values marked as ABCD, EFGH, IJKL and MNOP of the downloaded keyfile.



Download other needed resources
Below is a list of steps that need to be followed to have all the needed resources in place before running “tweetit”. Please note that all resources are hosted in github.
  • Create a directory called C:\tweetit and move the above created “keyfile.txt” there if not done before. Though you have a choice to place the keyfile.txt anywhere in your harddrive and refer the full path while running “tweetit”, it is prudent to place it under C:\tweetit\

  • Download tweetit jar application and save in the same directory location. The jar can be found here https://github.com/metallicatony/tweetit/blob/master/target/tweetit-1.1.jar?raw=true

  • As said before, tweetit picks the contents for your tweets from a spreadsheet (*.xls). The spreadsheet (.xls) should have the message content of your tweets and optionally it can contain the path to the images in your computer that you want to upload as part of your tweet. Please download a template of the spreadsheet from the repository https://github.com/metallicatony/tweetit/blob/master/src/main/resources/tweets.xls?raw=true and place it under C:\tweetit\

  • Download a sample picture https://github.com/metallicatony/tweetit/blob/master/src/main/resources/H4EAD_Banner_Woman.png along with the above spreadsheet and save it under the same directory C:\tweetit\

  • This has been already covered by the last step from the previous section. Double check whether you already got keyfile.txt after merging in your keys and tokens into it. This keyfile needs to go under the same directory C:\tweetit\
  • .
After the above steps are complete, you are now ready to bulk tweet using “tweetit”.

Tweetit Commands
Below are some sample commands that you can use from a windows command prompt to run tweetit. The same set of commands can be used if you are using from Linux or any unix variant operating systems too. The only change will be the referred absolute path in the command. Before you try out below commands, please navigate to tweetit’s home directory C:\tweetit\

Search & Retweet
The below command pulls a recent 250 tweets with hashtag #H4EAD and retweets the same. Kindly note that by default tweetit retweets the latest 500 tweets. The retweets are done every ~12 seconds
java -jar tweetit-1.1.jar --retweet --keyfile "C:\tweetit\keyfile.txt" --search "#H4EAD" --count 250 --every 12 

A screenshot of the above command in action



Clone Tweets
The below command pulls a recent 250 tweets with hashtag #H4EAD ,clones them and tweets them as new tweets to your timeline. Kindly note that by default tweetit pulls the latest 500 tweets but the tweets are done every ~12 seconds.
java -jar tweetit-1.1.jar --clone --keyfile "C:\tweetit\keyfile.txt"

A screenshot of the above command in action



Tweet using Spreadsheet contents
The below command pulls the contents of the spreadsheet that you downloaded and tweets them to your timeline. The path to the spreadsheet can be explicitly mentioned as below. If not, it assumes a default path C:\tweetit\tweets.xls
java -jar tweetit-1.1.jar --tweet --keyfile "C:\tweetit\keyfile.txt" --every 10 --spreadsheet "C:\tweetit\tweets.xls"

A screenshot of the above command in action



Tweetit is really helpful when you want to tweet and retweet bulk on a daily basis. It can save you a lot of time by automatically tweeting with a simple run of a command and thereby you can focus on your day to day activities without much intervention.

Wednesday, December 31, 2014

CRUD REST APIs using Java, MongoDB, Spring, Spring Data, Apache CXF, Maven and SLF4J

This project shows how to build a web application and perform basic CRUD operations using MongoDB as the datastore. It exposes the built operations as REST web services or APIs too. I created this primarily to show how to incorporate and kickstart MongoDB in web applications. And hence this post focuses more on MongoDB than anything else. I would recommend referring my previous posts if you would like to know the basics of building a web app CRUD REST & SOAP Services using Java and Apache CXF.

The following were used to build this project
1) Java 1.6
2) Spring Tool suite (STS) 3.1.0
3) Spring framework 4.0.0
4) Spring Data 1.6.1
5) Apache CXF 2.7.5
6) Apache maven 3.0.4
7) Log4j 1.2.16 and Slf4j 1.6.1
8) Jackson 1.9.0
9) SOAP UI 4.5.1
10) MongoDB 2.6.4

Objective
The objective of this project is to demonstrate a simple CRUD operation on a set of employee information in MongoDB. All the CRUD operations are exposed as REST webservices and so they can be invoked through any type of web client like SOAPUI or even using a simple web browser.

About MongoDB
Mongo DB is one of the most popular NoSQL database that persists information as JSON-style documents without enforcing a schema or a defined structure. This type of storage and retrieval is very helpful in types of applications where speed, performance and scaling overweigh data redundancy. The best documentation I have found so far is Mongo’s own official manual MongoDB Official Manual.

Install & configure MongoDB
Before proceeding to build the web app, it is required to install and configure MongoDB by following MongoDB Official Installation Instructions . A basic level of understanding about mongo concepts and commands are needed (MongoDB official CRUD operations) before proceeding further.
The next few steps are needed to prep MongoDB before proceeding to build the webapp. After installing MongoDB, let’s make MongoDB to NOT allow anonymous logins. And that's how a typical installation will be in an enterprise. This needs Mongo daemon/server to be started in “auth” mode. But before starting in "auth" mode, we need to create an administrator user.

1) Create administrator user via mongo console
> db.createUser({user: "administrator", pwd: "password", roles: [ {role: "userAdminAnyDatabase", db: "admin"}]});
Successfully added user: {
        "user" : "administrator",
        "roles" : [
                {
                        "role" : "userAdminAnyDatabase",
                        "db" : "admin"
                }
        ]
}

2) Create one more non-admin/regular user who has RW permission for all databases in MongoDB
> db.createUser({user: "metallicatony", pwd: "password", roles: [{role: "readWriteAnyDatabase", db:"admin"}]});
Successfully added user: {
        "user" : "metallicatony",
        "roles" : [
                {
                        "role" : "readWriteAnyDatabase",
                        "db" : "admin"
                }
        ]
}

3) Start Mongo daemon in “auth” mode
 C:\Program Files\MongoDB 2.6 Standard\bin> mongod.exe --auth --dbpath "C:\Program Files\MongoDB 2.6 Standard\data\db" 

4) Test whether Mongo allows anonymous operations. It won’t.
> show dbs
2014-12-30T10:08:17.669-0800 listDatabases failed:{
        "ok" : 0,
        "errmsg" : "not authorized on admin to execute command { listDatabases: 1.0 }",
        "code" : 13
} at src/mongo/shell/mongo.js:47

5) Authenticate with the non-admin user created in step 2)
> db.auth("metallicatony", "password");
1

6) Database “organization” and collection “employees” are something that will be used in our webapp. Switch to database “organization” and insert a new document in a collection called “employees”.
> use organization;
switched to db organization


> db.employees.insert({empId: 1, fname: "Richard", lname: "Stallman", deptName: "CS", salary: 5000});
WriteResult({ "nInserted" : 1 })


> db.employees.find();
{ "_id" : ObjectId("54a32c39961019d59eacd4ce"), "empId" : 1, "fname" : "Richard", "lname" : "Stallman", 
 "deptName" : "CS", "salary" : 5000 }

> db.sequence.insert({empId: 1});
WriteResult({ "nInserted" : 1 })

With that done, the installation and prep work of mongodb is complete. The next steps focus on building the web application that will interact with MongoDB. This basic web application will have the CRUD capabilities - create, read, update and delete employees from mongodb.

POM Changes
Spring Data is the backbone for all the interactions that are performed between the web application and the MongoDB. Spring data provides a consistent programming model and lets java programmers to use all the features of MongoDB using a repository style data access layer. To include spring data for mongodb as part of the project, the following additions are made to the application’s pom.

Add the version of spring data mongodb in the pom properties
1.6.1.RELEASE
Add the dependency under pom dependencies

  org.springframework.data
  spring-data-mongodb
  ${spring.data.mongo.version}


MongoDB connection properties & Application context configuration
The connection properties of MongoDB are stored in a file called "database.properties" and resolved when spring loads "applicationContext.xml" file.
  mongo.hostname=localhost
  mongo.port=27017
  mongo.username=metallicatony
  mongo.password=password
  mongo.databaseName=organization
The below property placeholder configuration from applicationContext.xml file loads the properties

Added to that, below are the bean definitions in applicationContext.xml file to instantiate a "mongoTemplate" that will be injected into Spring repository. The web application interacts with the Mongo layer using the repository.


 
 

 


    
    




 
 
 

Web application descriptor's (web.xml) context config location parameter points to the xml files that contain bean definitions. In our application, it points to both cxf-bean.xml and applicationContext.xml files. Below is a snippet from web.xml

  contextConfigLocation
    
     classpath*:cxf-bean.xml,
     classpath*:applicationContext.xml
    


Application Code Layers
The web app code is organized into 7 layers wherein every layer corresponds to a package. The layers are
  • Service Layer - contains the interface and implementation of the APIs exposed 
  • Request Layer - contains the API request entities 
  • Response Layer - contains the API response entities 
  • Adapter Layer - contains helper methods that can convert entity of one type to other 
  • Business Layer - lies in between service and repository layer and applies any business logics if any 
  • Domain Layer - contains domain entities, more specifically for this project, it comprises entities that represent mongo documents 
  • Repository Layer - the layer down in the stack that bundles all the mongo queries to perform the desired operation in mongo database 

Repository & Domain Layers
The layers other than repository and domain layers are pretty much self-learnable from the code. Moreover the code follows a similar pattern like that was discussed in the previous posts about building CRUD REST and SOAP APIs and building CRUD operations using Spring and Hibernate. The domain layer contains two entities - Employee and Sequence. The employee entity is the one that maps to any document in "employees" collection that was created in step 6. This is done by using @Document annotation. This is more similar to mapping an entity to a table record in a database table.
@Document(collection="employees")
Sequence entity maps to a special counters collection called "sequence". Every employee id is an auto incremented sequence number that is generated during run time. More about this can be read from mongo's official manual Creating auto incrementing sequence number . In our case, the helper method "getNextId()" handles the job of fetching the sequence document and returning the incremented sequence number like this
 /**
  * Gets the next sequence id
  * @return the sequence id
  */
 public Long getNextId() {
  // Get object id of the sequence document
  Query queryGet = new Query(Criteria.where("empId").exists(true));
  Sequence sequenceDocGet = mongoTemplate.findOne(queryGet, Sequence.class);
  log.info("objectId={}", sequenceDocGet.get_id());
  
  // Increment emp id of the document fetched above and return the new sequence number
  Sequence sequenceDoc = null;
  Update update = new Update();
  update.inc("empId", 1);
  Query query = new Query(Criteria.where("_id").is(new ObjectId(sequenceDocGet.get_id())));
  FindAndModifyOptions options = new FindAndModifyOptions();
  options.returnNew(true);
  sequenceDoc = mongoTemplate.findAndModify(query, update, options, Sequence.class);
  if (sequenceDoc != null) {
   return sequenceDoc.getEmpId();
  }
  return null;
 }
The code for repository layer resides in EmployeeRepository class. All the mongo specific operations are possible from this layer by spring injecting mongo template
@Autowired
private MongoTemplate mongoTemplate;
The repository layer comprises operations to
  • find all employees (this operation is exposed as REST API - HTTP GET /employees)
  • find all employees by last name (this operation is exposed as REST API - HTTP GET /employees?lname="name")
  • find employee by employee id (this operation is exposed as REST API - HTTP GET /employees/{empId})
  • create employee (this operation is exposed as REST API - HTTP POST /employees)
  • update employee (this operation is exposed as REST API - HTTP PUT /employees/{empId})
  • delete employee (this operation is exposed as REST API - HTTP DELETE /employees/{empId})
Most part of the code involved in performing the above operations is about creating the right Mongo Query and running it using the injected mongo template. Being comfortable with basic mongo query syntax will help a lot here. Feel free to download, build and browse through the code from my GitHub repository - CRUD APIs using Mongo.

Labels: ,

Thursday, June 26, 2014

A playful approach to tackle programming questions ("Phone words using dialpad" problem)

Programming questions are always tricky. It often makes you dazed and confused. Any software engineer interview for a programming position involves a couple of programming questions. After listening to a question from the interviewer, when you try to answer back and that's when you go erratic. That's pretty normal for everyone and there is no need to be worried and concerned. The objective of this post is to show a way to tackle such programming questions. It will illustrate a planned approach that you might want to tread on for easily arriving at solutions. I have taken a famous problem called "Phone words" to explain this approach.

“Phone words” is a programming question that features in many interviews to test the analytic skills of a programmer. The problem statement goes like this: You are given a 7 or 10 digit phone number. In every telephone or a cell phone, we have numbers ranging from 0 to 9 in the dial pad. Every digit in the dial pad has either 0 or 3 or 4 letters mapped for every digit. For the given phone number you need to find out all combinations of words that is possible using the letters of every digit of the phone number.

I couldn’t find a clean solution for this question anywhere and so wanted to post my solution along with the approach taken. Be it any programming question, if you work to find a solution without doing any ground work, it is not going to help at all. A step by step systematic approach is what will help you find a solution. It’s a skill to learn and acquire but an art to conquer!!

Programming problems can be addressed in the same way you prepare yourself to play a new computer game. Say, suppose you heard about a new game from a game-maniac friend of yours. He is in all praises for this game. And now, after listening to all those good stories from him, you are very desperate to play this game. You are restless. You wait with no patience for the day to arrive so that you can play this wonderful game. Imagine how you prepare yourself to play a computer game, a game that you have never played in the past. Imagine how excited you will be, sitting right before the screen waiting for the game to load. Once your game starts, your curious playful mind will begin to find out answers for the following questions

1. What exactly is the objective or mission of this game?
2. What are the aids and powers that I’m provided with as a player? Do I carry health tabs, armory, high-tech weapons?
3. Is there any specific learning or understanding that I need to do about my given aids and powers?
4. Can I play this game for the first few times just to familiarize myself without being mindful about my mission, without worrying about whether I’m winning or losing?
5. After I get a sense that I’m comfortable playing this new game, I have to think and plan a little bit around those deadly traps and devilishly humongous demons with whom I want to deal carefully. What is the appropriate time to use my powers so that they can be put to best use?
6. And finally, I will try to frame a strategy to tackle all my enemies so that I can effectively accomplish the mission of the game

That’s how you plan before playing a game, isn’t it? This exactly is the thought process that is needed to solve a programming problem. Once we are able to figure out answers for the above questions, it is going to be much easier to solve the actual problem. Let us follow the above steps and figure out how helpful they are in solving our “Phone words” problem.

What is the mission of this game?
The objective of the problem is to find out all possible words that could be created out of the given numbers. This is a classic example of a permutation and combination problem.

What are the aids and powers that I’m provided with as a player?
I’m given a set of numbers that represents a phone number and a phone dial pad that has numbers running through 0 to 9. Every number is mapped to a set of characters. Any one of those characters from every digit will be considered every time I try to create a possible word

Is there any specific learning that I need to perform about the aids and powers that I’m given with?
As a player, I’m provided with a phone dial pad and a set of numbers for which I need to create possible words. When I look at the dial pad, I see that not all numbers have the same 3 characters associated with them. It appears that there can be zero or more letters for a number. Digits like 0 and 1 don’t have any letters while digits 7 and 9 are mapped to 4 characters each. Wow, that’s a good point to understand before trying to solve the problem!

Can I play this game for the first few times just to familiarize myself without being mindful about my mission and without worrying about whether I’m winning or losing?
I want to play this game for a couple of times so that I can familiarize myself. I will be happy to choose the least difficult level so that it buys me more time on stage. Likewise, when a programming problem is given, it’s important to run through the problem with a couple of easy experiments. In this case, say for example, running through the problem using an imaginary phone number “23” will make me understand that the possible words are “AD”, “AE”, “AF”, “BD”, “BE”, “BF”, “CD”, “CE”, “CF”. Running through this exercise and illustrating the samples increases the confidence.

Can I think and plan a little bit around those deadly traps and devilishly humongous demons with whom I want to deal carefully?
When i'm playing a game, as soon as I spot some deadly traps, I want to think and plan a bit about them. I want to understand so that i can better tackle those traps. Likewise, there are a couple of traps in this problem. The numbers 0 or 1 do not have any characters associated with them. And there are numbers 7 and 9 that are mapped to 4 characters each, which is 1 character more than the typical 3 characters per number. This in turn means that a pure iterative solution might fail if it goes by the assumption that it is always 3 mapped characters per digit across board. I have to be careful about it. In case of 0 or 1, it will be the digits 0 and 1 used for creating a word and when it is 7 or 9, I must be careful to use all the 4 characters to create words. To re-familiarize myself, I need to run through those traps a couple of times to improve my understanding. Say, I’m now given with an imaginary number “702”. Then the possible word combinations will be “P0A”, “P0B”, “P0C”, “Q0A”, “Q0B”, “Q0C”, “R0A”, “R0B”, “R0C”, “S0A”, “S0B”, “S0C”.

Can i frame a strategy to tackle all the enemies so that i can accomplish the mission of the game?
Now comes the big part. I know quite a bit about the game now. I know about the powers that I’m given with, I know where to hunt for gold and I’m aware of the exact place and point at which those devils will show up. Now, it’s time for me to craft a plan, a plan that will defend me and defy all my enemies. Thinking at a very high level, there are two ways I can solve this problem.

Iterative Solution:
Let’s say I’m given with an imaginary phone number 234 ({abc}, {def}, {ghi}). Then as a first step, i have to think about the number of possible words that can be created out of that phone number. It will be 3x3x3 = 27 words.

Let me run through this solution in my mind. As the first digit of the given phone number is 2, I can have the letters - a, b and c stashed separately. Once I see the second digit 3, I can append every mapped character of 3 to the previously stashed words. Now the stashed words become ad, bd, cd and ae, be, ce and af, bf, cf. The next digit is 4 and so the stashed words will now become – adg, bdg, cdg, aeg, beg, ceg, afg, bfg, cfg, adh, bdh, cdh, aeh, beh, ceh, afh, bfh, cfh, adi, bdi, cdi, aei, bei, cei, afi, bfi, cfi. Running through this gives me a clue that this is an iterative solution. Additionally I need two lists, one with the stashed words and the other list for retaining the new words that are created by appending the stashed words with the letter that I’m iterating with. After appending all letters mapped for a digit with the stashed words, the new list becomes the stashed list after which the new list is emptied to ready for the next iteration. Simple, isn’t it?

/**
 * Returns the list of words created using phone dial pad and the given phone number
 * @param digits the phone number
 * @return list of combination of words
 */
public static ArrayList<String> getWordFromDialPad(String digits) {
  ArrayList<String> res = new ArrayList<String>();
  ArrayList<String> preres = new ArrayList<String>();
  res.add("");

  for (int i = 0; i < digits.length(); i++) {
   for (String str : res) {
    String letters = map.get(digits.charAt(i));
    for (int j = 0; j < letters.length(); j++)
     preres.add(str + letters.charAt(j));
   }
   res = preres;
   preres = new ArrayList<String>();
  }
  return res;
 }

This iterative solution looks simple but the only drawback is it needs more space to stash the list of words for every iteration. The complete implementation can be accessed from my GitHub repository - Phone words iterative solution.

Recursive Solution:
This is another solution for the same problem. In fact, this solution is a blend of recursive and iterative. The best part of this solution is that it just needs a single array equal to the size of the phone number!! Awesome isn’t it? No more extra space is needed!! Recursive solutions are interesting to understand and efficient most of the times. Instead of thinking iteratively, recursion will make you think differently. It wants you to go deep down and loop through all possible combinations in the lowest possible levels and then slowly climb upper levels to repeat the same. And that’s exactly this recursive method does
/**
 * Recursive helper method to create a word from dial pad
 * @param word character array
 * @param numbers given input numbers as string
 * @param start the index of input digit that is being processed
 */
private static void getWordFromDialPad(char[] word, String numbers, int start) {
  // base case -> start is the index value of the position from the word
  if (start > (numbers.length() - 1)) {
    System.out.println(++counter + " ==> " + new String(word));
    // look up dictionary to find whether the word is meaningful or not
    return;
  }

  // Identify the number of iterations based on the digit
  Integer number = numbers.charAt(start) - '0';
  int iterations = getSizeOfFirstDimension(number);


  // Iteration in a recursive method is equal to number of possible chars a digit can take
  // for eg. digit 2 can take 3 possible chars - A, B and C where as digit 7 can take 4 different chars
  for (int j = 0; j < iterations; j++) {
    // store a possible character for the digit and move on to the next position by 
    // calling the recursive method again
    word[start] = getCharacterFromDialPad(number, j);
    getWordFromDialPad(word, numbers, start + 1);
  }
}

That’s the core part of this recursive solution. The rest of the implementation has helper methods to support the above recursion. The complete implementation can be accessed from my GitHub repository - Phone words recursive solution.

A preface of any book will brief and summarize about what the book is about. Likewise it is very important to understand the preface of a programming problem by asking the above questions. It is very similar to the set of questions that you ask yourselves when you get ready to play a new game. Running through a couple of simple experiments will give deeper understanding and confidence. It will stand as a stepping stone to craft the final solution!!

Labels: , ,