Grok the Web

A Programmer's Guide to the New Software Development Paradigm

by Andrew Schulman

/grok/, var. /grohk/ [from the novel "Stranger in a Strange Land", by Robert A. Heinlein, where it is a Martian word meaning literally `to drink' and metaphorically `to be one with'] vt. 1. To understand, usually in a global sense. Connotes intimate and exhaustive knowledge. Contrast zen, which is similar supernal understanding experienced as a single brief flash.
-- from the jargon file


Introduction
Outline

The title "Harnessing the Web Software Revolution" would be fairly accurate, as the book describes how URL+HTML+HTTP+CGI is creating a major "paradigm shift" in software development, and shows the reader concretely how to employ this in their own programming. But that is a puffing, booster's title. The book also discusses problems with the web (the biggest one being how to make any money from simplicity, when what we're all set up for is making money from complexity), and the title should reflect this too.

This sounds kind of nebulous, I know, but basically this book will try to get the reader to see the implications (both good and bad) of the fact that we now have worldwide distributed computation, and worldwide compound documents, available with two lines of stuff that's barely even code. I'd say basically that, by the end of the book, the reader will "grok" the following few lines of code, in a way they hadn't before:

<HTML>
<A HREF="http://www.amazon.com/exec/obidos/ISBN=1568843054">
<IMG SRC="http://www.idgbooks.com/images/smallcovers/1-56884-305-4.gif"
ALT="Click here to order"> We
have an image from one machine; clicking on the image (or on this
text) starts up a process on another machine -- distributed
computation under the guise of hypertext!
</A>
</HTML>
This is a book whose goal is not just to give the reader some code (though it will do that), but more important to have the reader "grok" some concepts sitting behind code they may have already seen, but not pondered the implications of. The implications are huge.

Outline

1. The Crude Beginnings of a New Operating System

Using an in-depth look at a single example (a few lines of HTML to tap into amazon.com's database, while displaying a graphic from www.microsoft.com), the opening chapter shows how the web is in the process of revolutionizing software development: we now have distributed computation and compound documents, which were both supposed to be hard problems, in the form of hypertext/hypermedia links, which even a non-programmer can do. This is both wonderful and frightening. The URL concept is key.

2. An Address for Everything: URLs

(Includes sidebar on "ethics" of IMG SRC to original machine vs. IMG SRC to copied image vs. A HREF: "fair use".)

3. Compound Documents: HTML

4. Snarfing Forms

Starting from the importance of "View Document Source" for learning how to control/program the web, this chapter in particular shows how to go behind the HTML forms one sees every day, to create your own new forms. This requires a small but important conceptual leap: form and CGI aren't tied, there can be a form on one machine (yours), interfacing to a CGI on another. The author of an HTML form need not be the same as (or even know, or even have the permission of) the author of the underlying CGI. For this to work, you must uncover the "implicit API" provided by the CGI program. Because it's only implicit, this API can change at any time, so the chapter discusses brittleness/"bit-rot" problems too. A key point is the importance of <INPUT TYPE=hidden>.

5. Stealing Cycles

As suggested by the previous chapter, the web consists not only of hyperlinked documents, but also of processes which you can tap into. This chapter continues with this theme, but instead of turning forms into forms, we dispense with forms completely, and turn them (where possible) into URLs: distributed computation in the form of hypertext links! This requires understanding <FORM METHOD=get> vs. <FORM METHOD=post>. Using what seem at first like some "stupid web tricks" (hooking up NetCraft to the URL-minder), this chapter will show how to pipe one CGI process into another. That you can do this suggests that web sites are really tools, or software components. So perhaps a better chapter title would be "The Tools Approach to the Web."

6. The Document Is the Program

This chapter tries to make two different, but related points. First, "The content is the application": instead of huge monolithic applications such as WinWord or Excel, software development can now focus on writing things which really are applications, that is, programs which are tailored to very specific problems and markets. Since it's so easy to do the general stuff now (display dialog boxes, etc.), the focus of software development is moving rapidly towards the only thing that has genuine value: content, i.e., intellectual property.

The second point is that "the document is the program": Many pages which one sees on the web are, once you have the proper mind-set, not so much documents, as applications. The reader can use the page, not just read it, to get actual work done. The web turns the whole document/program relationship on its head.

This all sounds way too vague. Concretely, the chapter shows how HTML "documents" can contain all sorts of widgets for the reader to interact with: so they're not really documents, but programs. And with <SCRIPT>, these documents can now literally contain code, and parts of the document can be generated on the fly. These very specific points need to be tied back into the more general points the chapter is trying to make.

7. Web Browser as Display Engine

Suddenly dialog box programming (and user-interface programming more generally), which was supposed to be hard, is easy. Besides <FORM> and <INPUT>, another wonderful example is <TABLE>. You construct the HTML, and the web browser does all the work. The point, this chapter shows, is that what we call a web browser is really a *display engine*, an all-purpose application. (X-Windows and NeWs were great ideas; this is a workable implementation of the basic idea of separating an app from its UI.) You drive this display engine using a text file. This text file is written in HTML. (Text rather than binary means: no priesthood. This has both benefits and problems. In HTML, app drives the display engine at about the "right" level; contrast for example the lower-level X protocol or PostScript.)

I'm going to discuss my "local CGI" idea in this chapter: why require that a network sit between a CGI program and the web browser? Given that a CGI program can produce nice-looking UI, without getting involved in the micro-details of UI, why not make this same mechanism available on a single machine? The chapter probably needs to discuss Microsoft's "Trident" ("Dynamic HTML") here too.

8. Don't Write HTML!

Having sold you on the wonders of HTML as the lingua franca for the next generation of software development, in this chapter I'm going to tell you not to write HTML. Instead, write a program to write HTML. That of course is what a CGI program is, but this isn't really a CGI program: instead, it's a throw-away program to generate your HTML for you. Probably discuss benefits and problems to commercial HTML generators (HotMetal, FrontPage, etc.) here? Introduce the notion of "HTML compilers": programmers are not used to thinking of something as simple as HTML as "code," or to thinking of programs that generate it as "compilers." But behind the simplicity of HTML which is of course key to web's success, there's a lot of complexity in what you can do with it. HTML really is the code programmers are going to be dealing with the next 5-10 years. It will be the back-end produced by more and more programs.

9. Client/Server Made Easy

Until now, while we've talked a lot about CGI, it's been from a perpsective you don't normally hear about in discussions of CGI: the user's perspective. We haven't actually written any CGI apps. Instead, we've learned how to use <FORM> and <A HREF> and <IMG SRC> to make use of existing CGIs. In this chapter, we'll go around to the other side of the counter, and write some CGI apps. This requires understanding the client/server programming model. Client/server was supposed to be difficult, and the web quietly provided it, almost without anyone noticing (hence the chapter's title: not "made easy" in the sense of "for dummies 101" but in the sense that web once again quietly provided something that the previous software-development paradigm had said was incredibly difficult).

10. "Activate the Internet"?

What about ActiveX and Java? Where do they fit into all this? Microsoft tells developers to "Activate the Internet", and Sun implicitly says the same thing. Yet in the previous chapters we've seen that the web already is active, thanks to such ridiculously simple mechanisms as CGI and perl. Perhaps 75% of what developers think they want ActiveX or Java for, they could be doing with CGI. But what about the remaining 25%? How to use Java and ActiveX without suddenly returning to the bad old days of overly-complex software development? How to employ these sensibly, where they make sense, without infecting the vas majority of situations where they don't make sense? Beats me! (Hopefully I'll have a better answer to these questions by the time I get to chapter 10.)

11. "The Network is the Computer"

Sun's slogan "the network is the computer" has an interesting implication: if more and more of what one does on a PC involves the web browser, if what's on the web becomes more interesting than what's on your local hard disk (and is certainly easier to find!), then what becomes of Windows? The requirements of both HTTP client and server from the underlying operating system are fairly minimal. (We saw this in the NT Workstation vs. Server episode.) So shouldn't we be evolving away from huge OSes like Windows, towards much simpler "thin" operating systems (back to the ultimate "thin" OS, MS-DOS?!) and machines? The ultimate expression of this is WebTV, where you can do web browsing and email with a TV-like remote control, and where the machine costs $400 instead of $3000. Benefits and problems to this approach, but possibly major implications for both hardware and software manufacturers. What does Web really mean for Wintel?

12. HTML Everywhere

In chapter 7, we saw that what is called a web browser is really more general purpose: a display engine. This display engine is driven with a text-based protocol called HTML. In this final chapter, we see the wider implications of this: HTML for email, for word-processing and spreadsheet documents (Office 97: at what point does HTML become the "native" file format for all these "productivity" applications?); for remote system administration (web "gateways" to Unix tools), controlling embedded devices (see smallest.pharlap.com), etc. The chapter title takes off on Microsoft's old slogan "Windows Everywhere." ("Local CGI" fits in here too.)

Boxes

Noticeably absent from this outline are three key web problems: security, performance, and how to make money. Rather than being given separate chapters, these will be discussed throughout the book, in boxes/sidebars. For example, "Rich Text and Email Bombs" in chapter 1.