The state of the Internet operating system
1

The state of the Internet operating system

Tim O’Reilly offers a definition of the Internet operating system:

This is the crux of my argument about the internet operating system. We are once again approaching the point at which the Faustian bargain will be made: simply use our facilities, and the complexity will go away. And much as happened during the 1980s, there is more than one company making that promise. We’re entering a modern version of “the Great Game”, the rivalry to control the narrow passes to the promised future of computing. (John Battelle calls them “points of control”.) This rivalry is seen most acutely in mobile applications that rely on internet services as back-ends.

This is a great and important article, an explanation of where we are right now as an industry.

The two questions that I’m interested in trying to answer, or seeing other people try to answer, are:

  1. What are the keys to preserving the freedom and openness of the Internet given the trends in the article?
  2. How should people developing applications using the Internet operating system be investing their time right now?
Building a new service from scratch, followup
0

Building a new service from scratch, followup

About a month ago I asked what people would use to build a new Web service from scratch, without providing any real details on what the application would do. People seemed to all agree that basing the service on JSON was the best way to go, and that’s the route I took, so I thought I’d write it up.

In this case, I have a PHP front end talking to a Java back end that uses Spring and Hibernate. The application in question is the administrative console for a test harness for a SOAP Web service. We use a third party SOAP service and needed the ability to provide responses that are guaranteed to always be the same for the purpose of creating tests. So my test harness stands in for the third party service, requiring only a URL change, and returns the data managed via the client application to which my question referred.

The application itself is very simple — it’s a single table with a lookup mechanism. The client enables users to find and edit individual records from the table. I needed four basic operations:

  • Search for records
  • Retrieve a single record
  • Update a record
  • Create a record
  • Delete a record

The Server

This is a Java application that uses Spring and Spring MVC. I use Spring’s Web Services support for the SOAP stuff, but it wasn’t worth it to bother with that for the administrative client. So instead I mapped every URL ending in .json to a Spring DispatcherServlet. Like all sensible Java developers who’ve worked in Rails, I hate configuration in general and XML configuration files in particular. So I used Spring’s ControllerClassNameHandlerMapping, which extracts the path from the URL and maps it to a Spring controller.

So a path like /myapp/responses/search.json calls the search() method in the ResponsesController class. Simple.

I won’t go into the boring details of my implementation, other than to talk about how I’m passing the messages back and forth. The client uses standard form submission (name and value pairs) to submit the data, and expects to get JSON back. So in the case of search, I extract the field name and query value, go to the database and run a query to retrieve the records in question, and then package them up and encode them using JSON to be sent back to the client.

The JSON encoding is handled by Google’s GSON library, and it’s ridiculously simple. It accepts a plain Java object, and serializes it and all of its children using reflection. Any properties that are null are omitted. So for my application, I created a wrapper object that was well-suited to being encoded using GSON. It includes a hash for error messages, a property for a single response object, and a collection of response objects for search results.

Then I created a view for Spring called SimpleGsonView, which accepts a wrapper (or any object), encodes it using GSON, and then sends the results to the client. Here’s the whole thing:

@Override
protected void renderMergedOutputModel(Map aMap, 
    HttpServletRequest aRequest, 
    HttpServletResponse aResponse) throws Exception
{
    Object wrapper = aMap.get("wrapper");
    Gson gson = new Gson();
    aResponse.setContentType("application/json");
    ServletOutputStream out = aResponse.getOutputStream();
    out.print(gson.toJson(wrapper).replaceAll("\\\\n", ""));
    logger.debug(gson.toJson(wrapper));
    out.flush();
}

One thing you may notice is that I do a search and replace on the JSON. It removes all occurrences of \n from the encoded data. That’s a hack to get around a client side bug that I got tired of trying to fix. I’ll talk about that later.

So for this application, to add new functionality, I just add a new method to my controller, process the input parameters and put the correct results in the wrapper, and then pass that to the view I created. Very simple. I added an autocomplete feature to the search page, and this is all I had to do to add support for it on the back end:

public ModelAndView complete(HttpServletRequest aRequest, 
    HttpServletResponse aResponse)
{
    String searchField = aRequest.getParameter("field");
    String searchQuery = aRequest.getParameter("query");

    List<String> results = Collections.emptyList();

    if (searchField.equals(FIRST_NAME))
    {
        results = getFirstNameCompletions(searchQuery);
    }
    else if (searchField.equals(LAST_NAME))
    {
        results = getLastNameCompletions(searchQuery);
    }
    else if (searchField.equals(KEY))
    {
        results = getKeyCompletions(searchQuery);
    }
    else if (searchField.equals(ZIP))
    {
        results = getZipCompletions(searchQuery);
    }

    return new ModelAndView(new SimpleGsonView(), "wrapper", 
        results);
}

The methods to retrieve the data are in another class, but I removed the reference to it for simplicity.

So that’s the server piece. I could have done more to make it RESTful but since the service is just for me, I went with the simplest possible approach.

The Client

For legacy reasons the client is a PHP application, and I decided to use jQuery and a little PHP to implement the client features. Here’s how each operation is implemented:

Search: The search feature is pure jQuery. The home page for the client has a search form on it with a pulldown of fields that can be searched and a text box for the search terms. I used the autocomplete feature from jQuery UI to make the search form a bit friendlier. I catch the submission of the search from (or the selection of an item in the list provided by autocomplete) and submit the results to the server using jQuery’s getJSON() method. Then I create a table on the page with the search results in it.

Retrieve a single record: This is the only place I used PHP to connect to the server. I decided that I wanted each item to have its own URL, and although I could have written the code to pull the key for the item in the URL, fetch it, and build the page in JavaScript, I opted to use PHP instead. To do so, I grabbed a copy of the Services_JSON library and included that in my app. PHP supports JSON natively starting with version 5.2, but my server is running PHP 5.1. I chose Services_JSON because it’s a single file. So when you go to the edit or show page, I use PHP to fetch the JSON for the key in the URL from the server and put it on the detail page or in the edit form. This is where the bug comes in. My service returns XML documents (encoded as a value in a JSON hash), and sometimes Services_JSON won’t decode the data properly. When I started stripping \n from the XML, it worked, so I went with it.

Create a record: This form uses jQuery’s post() method to submit the contents of the form to the server. If the request returns a successful result, it redirects the user to the detail page for the new item. If it doesn’t, it displays the errors included in the result.

Update a record: This form works just like the create form.

Delete a record: This is the simplest operation of all. It submits the key to be deleted, and the server simply returns true or false.

Upshot

This was just about the easiest implementation of an application and accompanying Web service I’ve ever done, in spite of the fact that I’m a pretty terrible JavaScript developer. The server side was much easier to implement than I’d anticipated, it took less code in Java than it would have in PHP, and not much more code than it would have taken in Ruby. This is the application that really convinced me that there’s essentially no development overhead involved with using AJAX rather than traditional server-side request processing when building Web applications, at least if you bring libraries like jQuery and GSON to the party to make things easier.

Building a new service from scratch
13

Building a new service from scratch

I’m building a very thin Web interface for an internal application. The back end is built using Java, and I’ll be providing a Web service for use by the front end, which is built using PHP and JavaScript. The question is, if you were starting from scratch today, what would you use to glue the two together?

Options include:

  • Providing a SOAP service, and implementing the SOAP client in PHP. (This option is only included for completeness, I’m not going to implement it this way.)
  • Providing a REST Web service that returns XML and implementing the client in PHP.
  • Providing a REST Web service that returns XML and implementing the client in JavaScript.
  • Providing a REST Web service that returns XML and using JavaScript and PHP as the client where appropriate.
  • Providing a REST Web service that returns JSON and implementing the client in JavaScript.

I have been leaning toward the JSON solution, but I’m curious to know what other people think.

Simon Willison on the Flickr API
3

Simon Willison on the Flickr API

Over at the Flickr developer blog, Kellan Elliott-McCrea interviews Simon Willison about how his new project, Wildlife Near You, uses the Flickr API. Simon explained how the site came to be here — it was built by a team of geeks who went to an exotic locale to see what they could build in a week. The interview is a great read if you want to get jazzed about Web services — providing a robust machine interface to a site’s functionality can really unlock a ton of power.

Wildlife Near You is definitely worth browsing as well, just as a primer on what can be done quickly these days if you have a lot of talent and you’re starting from scratch. I particularly love the lists of species found on the place pages. They’re the most aesthetically pleasing and useful tag cloud implementation that I’ve seen.

Update: I left out the link to the interview itself.

Links for August 27
2

Links for August 27

  • Simon St Laurent looks at reasons why there’s buzz around HTML again.
  • Mac OS X Automation explains Services in Snow Leopard (my copy arrives tomorrow). Via Daring Fireball.
  • A new poll reveals that people don’t actually even know what the public option is. The public option is a government-managed insurance plan that will compete with plans from private insurers in an exchange, available to individuals and small businesses that do not participate in group insurance. Here’s a longer explanation. In the meantime, the current Republican talking point seems to be that Medicare is a poorly run government program that we should preserve at all costs.
  • The MySQL Performance Blog looks at the Redis database. Redis is one of those schema-less databases people are all talking about these days.
  • Matt Raible takes a look at Java REST frameworks.
  • The UK is looking at plastic alternatives to traditional pub glasses. That wins my “stupidest thing I read today” award. Via Bruce Schneier.
The New York Times’ Congress API
1

The New York Times’ Congress API

The New York Times has released a new, free Web service that developers can use to create applications that track how Congress members vote. Join me in offering kudos to the New York Times and in wondering why this service isn’t provided by the government itself.

If I were named CTO of the United States government, I’d require that all new public-facing government applications expose Web services, and that internal developers use those services to build their own applications, thus insuring that those Web services remain useful and up to date. It would both improve transparency and probably increase the quality of the applications offered by the government.

A Web 2.0 perspective on open government
4

A Web 2.0 perspective on open government

Ed Felten and some co-authors have written a paper discussing a more modern approach to open government. They suggest that the best approach is that rather than government agencies focusing on building Web sites that provide access to government data, they should focus on providing the data itself in an open format so that it can be utilized by anyone.

Here’s the meat of the proposal:

Rather than struggling, as it currently does, to design sites that meet each end-user need, it should focus on creating a simple, reliable and publicly accessible infrastructure that “exposes” the underlying data. Private actors, either nonprofit or commercial, are better suited to deliver government information to citizens and can constantly create and reshape the tools individuals use to find and leverage public data. The best way to ensure that the government allows private parties to compete on equal terms in the provision of government data is to require that federal websites themselves use the same open systems for accessing the underlying data as they make available to the public at large.

Sounds like a great idea to me. I love the idea of forcing government agencies to eat their own dog food in order to propel this kind of approach.

Links for April 13
1

Links for April 13