• #urllib #rest #api

In an age of web services being able to interact with them is almost a necessity for any developer. Unfortunately, if you are a new’ish python developer it might be a bit hard to figure out how to do it since there is no straight forward way. So in this post I will show you some of the basics by doing a GET and POST, and showing the nuances.

This blog post assumes you know the purpose of GET and POST HTTP verbs.

Getting Acquainted with urllib

Urllib is a built-in python library which allows you a, fairly, easy way to interact with remote resources. The problem is if you don’t know how to use it, at first, it can be confusing.

There are two main methods you need to be concerned with:

urllib.urlopen()
urllib.urlencode()

urllib.urlopen is what you will actually use to call a web service. While urllib.urlencode is what you are going to use to prepare your data to send to your web service. Those two functions are really all you need to know to get started.

GET Without Data

Sometimes all you are needing to do is get data with no special filters or anything so all we need to do is call urlopen and give it a url.

import urllib
u = urllib.urlopen("http://localhost/users/all/")

print(u.read())

Lets take a walk through the code:

import urllib

This imports our library for use. Fairly basic stuff.

u = urllib.urlopen("http://localhost/users/all/")

Here we are actually calling our web service and assigning an instantiated IO type object similar to python File object. This gives us a few options how to deal with our call, and data. Since we are going to keep this basic we are going to just look at one method.

print(u.read())

This is where the fun is since this dumps all the data we got back from our web service to console. Here you would generally apply some sort of deserializer to get the data in a more programmatic format. I usually deal a lot with JSON so you would want to decode the JSON and deal with it from there.

Now that we have an idea of what we are doing lets look at a couple of different variations to how we call because that determines what urllib does.

GET With Data

Sometimes we need get data, but it needs a filter on it so we need to send a bit of data along with our get request.

data = urllib.urlencode({"contains":"buddy"})

u = urllib.urlopen("http://localhost/users/all/?{}".format(data)

With line one we urlencoded our dictionary which turned the dictionary into a query string type syntax we then just added to the end of our urlopen call. So what urlopen method really called was “http://localhost/users/all/?contains=buddy”. The good thing about doing it the way we did is in case there are special characters which need to be encoded it can take care of it. Also beats the heck out of manually building a long set of query strings.

POST Data

The final thing on our list of what we are doing is POSTing data to a web service. This is done the exact same way, except with a few minor character differences. Lets take a look.

data = urllib.urlencode({"name":"buddy lindsey","height":"5 foot 5 inches"})

u = urllib.urlopen("http://localhost/users/create", data)

If you look at the code the only thing that really changed was I added more data, and the urlopen call is a bit different. Instead of adding data as string to the end of the url it is passed as a parameter to the urlopen method. This tells urlopen “hey use a POST instead of a GET”. Nice fun nuance isn’t it.

Conclusion

It is amazing how something so simple can be so convoluted when you start. The first time learning it it isn’t very simple so I hope this is laid out well. There is much more to learn to do more advanced “stuff” with web services, but this will get you most of the way there dealing with them, without a 3rd party library.