The V6 Engine

Ghost Rider

Bernard Lang and François Rouaix
March 14, 1996

Presented at the WWW5 Workshop: Programming the Web - in search for APIs
Paris - 6 May 1996

Abstract : WWW browsers may be seen as the composition of a multimedia interface, and a set of internet communications facilities. We propose (like some other authors) to extend the communications/processing functionalities by means of filters placed on the communication lines. The originality of this proposal lies in an architecture to efficiently compose filters, and in a simple use of the HTTP protocol, without extensions, to control the configuration of the set of filters, as well as the configuration of services embedded in each filter. We list a number of applications that show that existing communication protocols are sufficient for a very wide range of browser enhancement, with little need for other APIs.


The client-server model has been most successful in its World-Wide-Web (WWW) application to internet. A major drawback of this success is that, since Web browsers are fast becoming the unique/universal tool for accessing all services on internet, they are also becoming huge and unwieldy general purpose pieces of software.

One reason for this success is that for a large number of applications, the browser paradigm has proved a rich, effective, and friendly man-machine interface, easily understood by the average user, which tends to make it a unique standard interface whenever possible.

This is so true that some developers have started using Web browsers and the hypermedia facilities they offer to produce cheaply friendly man-machine interfaces for non-Web applications. The development of some distributed applications has lead naturally to this conclusion: these applications are installed as CGI applications on some server, which mediates HTML interfaces with browsers, using forms for user input [ Harvest, SATAN ].

This cannot usually be done for an application running on the user workstation (or personal computer) since that station will not in general run a server. However, with very minimal addition to a browser, it is possible to use it as an interface to CGI-like programs run on the local machine. This feature is available on the MMM navigator. MMM allows the user to specify a "local binaries directory", say /home/WWW/local/bin such that any URI of the form file://localhost/home/WWW/local/bin/foo?...arguments... will be considered as a call to a program foo in that directory. The text output of the program (usually HTML source) is then displayed by the browser, spiced with links to documentation or to further requests. This approach was used to implement very cheaply CD-Punch, an interactive debugger (still a prototype) for ISO-9660 file systems, with full online documentation.

Our conclusion is that Web browsers have 2 aspects:

Coming back to our original concern that browsers are becoming unacceptably large, the obvious conclusion is they should be structured into smaller more manageable modules. Indeed this trend is already visible since much of the growth of browser capabilities is achieved by linking independent extensions through (yet to be) standardized APIs. However this still maintains a fairly monolithic runtime structure revolving around the chosen browser.

Another issue to be considered is that we should expect browsers with various capabilities to appear on the market, and that, most likely, users will have several browsers available, more or less convenient, depending on the type of site or data they want to access. In such a context, it should be made easy to switch between browsers to observe a given site. This will in particular require that much of the communication data used by one browser be made easily available to others.

As was already hinted by our conclusions above, the easiest solution to achieve this is to separate in browsers the display functionality from the internet communication functionality. This will at the same time allow for a much greater flexibility in configuring internet navigation systems while simplifying the architecture, and reduce the browser size problem by breaking it into two fully distinct components with more specific purposes. Hopefully, this architecture will allows software producers to concentrate their efforts either on multimedia interface capabilities, or on internet communication capabilities, and thus will let the users build their configurations from the best products of each world. The separation of concerns may also clarify the definition of APIs.

We present below a browsing environment/architecture which aims at separating Display from Web functionalities. In order to stay compatible with the existing technology, our design assumes that the multimedia interface capabilities are provided, with no change, by the already existing browsers, though we suggest some minimal change to make the architecture more powerful. We then discuss a variety of applications that are allowed by this architecture.

The web communication management and enhancement is realized by a proxy like program that acts as a filter between the client and the server. Such use of filters has been proposed by several people, usually for specific communication or information management enhancements [ WebFilter, Fast Forward, Annotation support ]. This approach has been systematized in the collection of services developed by the OSF WAIBA group [ Stream transducers ], who is also proposing a scheme to organize the cooperation of these filters [ Filter integration ]. The latter proposal by the OSFF WAIBA group is closest to our in functionality though different in realization. OSF proposes to organize each functionality as an independent transmission agent on the HTTP communication path, while we suggest modularly grouping functionalities into larger entities, around a support backbone, the V6 engine, which is the object of this paper.

The first version of the V6 Engine, that implements the ideas presented in this paper, has been programmed in the CSL version of the CAML functional programming language, which in particular offers threads to program the parallelism needed for such applications. It has been so far populated with a few simple components, and tested with both the Netscape and MMM browsers, on the Linux and SUN-OS operating systems.


The basic idea of the architecture is to deport net related facilities of the browser to an independent program, the V6 engine, which in first approximation is much like a proxy, i.e. it mediates invisibly the communication between the browser (client) and the servers. However, unlike usual proxies that are supposed to be essentially tranparent communication optimizers, the V6 engine has the following features: Actually, though we consider that a primary use of V6 will be as a functionality extender for personal browsers (and it was originally thought of for this purpose), it can also, like proxies, be used to implement shared functionalities for a group of users. In this latter case, the owner is the V6 engine administrator. Whether the tuning of the code will have to depend on the intended number of users will be a topic for experimentation. However, assigning a instance of V6 to a single user will simplify and better ensure the dynamic control of configuration and throughput. Furthermore, it is not clear to us at this point whether there is a significant number of applications requiring otherwise.

[V6 Icon] [ V6 architecture diagram ]

Though V6 appears like a proxy, and is declared as such to your browser, it can actually be attached to, and accessed through several ports . The ports may differs by the protocol used (it needs not be HTTP, though that is the only protocol currently implemented), but also by the way the information will be processed, which depends on the sequence of filters attached to the port.

A V6 HTTP port may be declared as HTTP proxy to any standard web browser. The V6 user configures his instance of V6 by attaching to this port a sequence of query filters . Each query that enters the port is processed by each query filter in turn. The filter may leave it unchanged or alter it in any way it sees fit, including changing header lines, and/or modifying the requested URL. At the same time it processes the query and passes the result to the next filter, the query filter creates () a reply filter , which may depend on the query. The reply filters are stacked together and will be used to process the answers to the query.

After the last query filter has been processed, the query is passed to the V6 Distributor which, depending on the URL requested, will pass it to one of several V6 services (their type and number is of course configurable and extensible). A virtual host name (we could have several), say "service.v6", is assigned to the instance of V6 being used, that should not conflict with any host name on the network. If the URL passed to the distributor start with the V6 address "http://service.v6/", then the request is to be processed locally by one of the V6 service modules. Otherwise it is transmitted via the HTTP client service if is is a HTTP request, or via the Proxies service if it is another protocol (FTP, Gopher, NNTP, ...) since the (current) implementation of V6 will not handle directly these protocols, though it can process queries/replies using them encoded in HTTP. The HTTP client can either run through the usual shared proxies, or access directly HTTP servers. The Proxies service passes the request to appropriate proxies that can handle the non-HTTP protocols used to communicate with servers.

If the URL is for a local processing, say for example "http://service.v6/cache/", then the distributor checks whether a service is attached to this URL. If yes, the query is transmitted to this service, which produces the reply. Otherwise, an error reply is produced by the distributor.

Replies to a query, whether coming from the network or from a service module, are passed to the stack of reply filters that was created when processing the corresponding query. Again the filters may modify the header of the reply or the reply itself in any way they see fit. After the last filter, the possibly transformed reply is passed by the port to the browser, or any other application program, that emitted the original query.

Actually the internal architecture is somewhat more complex, in order to deal with some technical problems, for example the control of the amount of parallelism, or the limitation of the buffering between reply filters. But the V6 users need not be aware of this. Actually, in most cases, they need not be aware of much of the above description, unless they want to write their own filters. The casual user will simply use the service module Services to include preprogrammed components on her V6 backbone.

Local URLs, and virtual hosts

There are actually several ways of adressing V6 services. One is by means of its virtual host name like "http://service.v6/" used above. The other is by using the URL for entry ports, for example "" if the port used to communicate with V6 is 1515 on host "". This URL is tranformed into a local one when the query passes the port.

More generally, one can include filters that can change an URL into another one, either local or global. This may be applied in many ways. One is the caching service described below.

Building a body around V6.

V6 is only an engine that serves as an infra structure to run a variety of modules, filters and services, that control the client-server communications. To illustrate the power and flexibility of the architecture, we describe a few applications already implemented.

The service Services

When V6 is activated for the first time, it comes with a minimal configuration that includes essentially the service Services at the URL "http://service.v6/services/". As with the Harvest system, the user can access at any time this URL with her favorite browser, and receives a page (actually several pages linked together) that gives her the current configuration, and allows her to install (or remove) new filters, and new services with associated URLs.

From this page, the user can access specific services (they can of course be accessed directly with their URLs) in order to check or modify their configuration.

The V6 distributor keeps a list of virtual URLs attached to the various services. When a new service is added dynamically, it is registered with the distributor for the URLs it serves. The distributor is also informed whenever the configuration of a service is changed.

The services: Files and CGI

Two standard services are Files and CGI which perform essentially the usual function of a server, i.e. to map URLs to files or executable programs of the local host. In this context, CGI programs are local programs that may be controlled through a browser interface.

User authentication

Modules of the V6 engine can manage and store confidential information, such as site passwords, cached pages of confidential data, etc. They may also give access to private files that the owner has remapped to private URLs. And, of course, they control the configuration of V6 itself.

It is therefore essential to make sure that only the rightful owner can access confidential information or services. The basic proxy authentication proposed in HTTP 1.1 has been implemented in an Authentication filter (alternative such filters could be made available, based on different authentication mecanisms). If the request fails the authentication check, the query is changed into a local query addressed to the Denial service, which will produce a HTML reply denying the access, and possibly asking the user to authenticate herself (with a form).

A more lenient approach, though harder to manage, is to allow access to V6 services that are not security risks to unauthenticated users, by checking the requested URLs. Different forms of access and authentication may be attached to different ports.

Caching service

A cache has been implemented by combining a filter and a cache service that runs the cache memory. When the query reaches the cache filter, it checks whether the requested URL is in the cache. If there is a hit, the URL is changed to "http://service.v6/cache/get?document-index" so that the query is passed to the Cache service rather than being forwarded outside. The cache then answers with its copy of the requested page. Otherwise, the query is transmitted normally.

In case of hit, the cache filter produces a reply filter which does nothing. Otherwise, the reply filter produced will make a copy of the reply in the cache memory.

Of course, this scheme allows to manage several caches with different policies. In particular it is possible to have specialized caches, that will be chosen depending on the URLs being addressed, using PICS ratings for example. One can also cache documents that are often not cached by browsers, in particular those accessed with other net protocols or services.

Other applications

Developing all the possible applications is beyond the scope of this position paper. We will briefly sketch a few of the areas we are considering:

V6 APIs and suggested extensions to existing browsers

The whole purpose of this work is to increase the modularity of web browsing tool, and to enhance existing ones without changing them. This is a rather simple approach to the API problem, when it can be worked out with existing communication protocols.

Though the extension modules (filters and services) are currently written in CAML, a system interface will be provided to allow filters written in any language. New services can already be implemented in any language with the CGI protocol. The communication protocol for filters is just a pipe mechanism for HTTP headers and data.

With respect to existing browser, one feature that would be useful is an API to create new buttons, or new items in a menu, that would be used to activate processors on the page being viewed. Even better, from our point of view would be the possibility to request (or later delete) a new button via the HTTP protocol (as part of a document header, for example), and to attach to it a URL which would be accessed with the current page URL as query string when the button is depressed.

Discussion and conclusion

The salient aspects of this proposal are: This architecture is somewhat different from that proposed by the OSF WAIBA group. Their decision to use of separate filters, though it leaves the components independent, raises many communication problems related mainly to connectivity consistency and communication overhead. These problems do not arise (or are lessened) within V6 support structure, while the modularity (of Strand-like fillings [ Filter Integration ]) is retained. The separation of filters and services actually increases the modularity, since services can be both shared by different filters and reconfigured dynamically. Typically security of access to confidential data in the filters has to be checked only once, when accessing the V6 engine, rather than every time a filter is entered. If there is a need to separate filters, for example to run them on distinct machines, or share some of them within a group of users, it is possible to connect several V6 engines as one would simple filters: the V6 engine may be seen as an efficient filter composition device, but does not preclude the OSF WAIBA approach. The communication with filters for configuration purposes is done exclusively with virtual URLs, without modifying the HTTP protocol or the pages returned by servers, even when filtering other protocols than HTTP. This discussion applies to call-return communication protocols like HTTP. Browser extensions using a different communication paradigm have not been considered and may require a different approach.

An essential point in this presentation is that net related functions may be "physically" separated from multi-media display functions, thus avoiding to mix concerns related to two distinct technologies, and allowing developpers to concentrate their effort on one problem only. Furthermore, one can avoid any need for new APIs other than the existing protocols, though slight enhancement of browsers might help.

A possible conclusion of this work is that, where management and processing of distributed structures is concerned, protocols for distributed communications should be preferred to APIs, though this should be further confirmed by more extensive experimentation with V6 or similar systems. Of course this is probably not true for the integration of multi-media facilities, which may be another reason to keep separate these two types of concern.


Disclaimer : The V6 Ghost Rider is in no (high)way other than HTTP connected to Roadkills-R-Us,
B.P. 105
78153 Le Chesnay CEDEX
tel: +33 (1) 3963 5644 and 3963 5629

Last modified: Friday, May 3, 1996