.. _rfc16:

=================================
MS RFC 16: MapScript WxS Services
=================================

:Date:  2006/05/10
:Author: Frank Warmerdam
:Contact: warmerdam at pobox.com
:Last Edited: May 22, 2006
:Status: adopted and implemented
:Version: MapServer 4.10

Purpose
--------

The general intention is that a WMS or WCS service should be able to 
be setup via MapScript.  The web request would be turned over to a 
MapScript script (Python, Perl, Java, etc) which can then manipulate
the request and a map as needed. Then it can invoke the low level MapServer
request function (ie. GetMap, GetCapabilities), and if desired manipulate
the returned information before returning to the client.  This will provide
a means to do customizations such as:

* Implement security policy. 
* Use dynamically created maps from a database, etc. 
* Flesh out capabilities documents with auxiliary information not settable
  via the .map METADATA tags. 
* Adjust error behaviors. 


Technical Solution
-------------------

* GetCapabilities, GetFeatureInfo and GetMap calls for WMS callable from 
  MapScript, and results capturable for processing.

* GetCapabilities, DescribeCoverage and GetCoverage calls for WCS callable
  from MapScript and results capturable for processing.

* GetCapabilities, DescribeFeatureType and GetFeature calls for WFS callable
  from MapScript and results capturable for processing. 

* Any other OWS services dispatched through OWSDispatch (such as SOS) would
  also be accessible from MapScript. 

* IO hooking to capture various output from MapServer services will be
  accomplished via mapio.c services, the same as is used to capture output
  for FastCGI services. 

* All SWIG based MapScript languages will be supported (Perl, Python, Ruby,
  C#, Java).  PHP (non SWIG) may be supported if the PHP MapScript maintainers
  do a similar implementation. 

* A MapScript WxS HOWTO will be written, including simple examples of 
  customized services. 

WxS Functions
-------------

Add the following methods on the mapObj in mapscript/swiginc/map.i.

::

 int OWSDispatch( OWSRequest *req );

We can't call the lower level functions, like msWMSGetCapabilities() 
directly very easily because these functions require some pre-processing 
done by msWMSDispatch().  

* Note that the OWSDispatch() reconfigures the map it is invoked on to match 
  the request.  
* Note that the results of the OWSRequest are still written out via the 
  normal stdout stream handling, so separate msIO hooking is needed to capture
  the results. 


OWSRequest
----------

This object is already defined to MapScript in mapscript/swiginc/owsrequest.i
but it seems to lack a means of setting it from cgi arguments or directly
by parsing a provided url.  I propose to add the following method on the
OWSRequest:

::

 loadParams();

* Loads the parameters from the normal sources (QUERY_STRING env. variable
  for instance).  

::

 loadParams( const char * url );

* Loads the parameters from the given url portion as would have appeared in 
  QUERY_STRING if REQUEST_METHOD was GET.  

  
IO Hooking
----------

Currently output from functions such as msWMSGetCapabilities() is directed 
through the msIO services (in mapio.c), such as msIO_printf().  In order to 
capture this output and make it available for further processing it will be 
necessary to provide a means for MapScript applications to insert their own 
IO handlers of some sort.  

Additionally, currently the msIO layer has a concept of io handlers, but
they are global to the application.  For non-trivial use of WxS MapScript
services in Java, c# and other multithreaded environments it is desirable
for it to be possible to insert per-thread IO handlers. 

* Need to make at least current_stdin/stdout/stderr_context variables thread 
  local.  Possibly using the same approach as maperror.c.  A new mutex will 
  be required for this. 
* Consider thread safe output to shared stdin/stdout/stderr handles for all
  threads? ie. protect with a mutex.
* We need to provide a convenent way to install "to buffer" and "from buffer"
  io handlers. 
* We need to always use msIO redirection.  Currently in the default case of
  not using FastCGI, USE_MAPIO is not defined in mapio.h and msIO_printf()
  and similar functions are actually just #define'ed to printf().  But if
  we want to be able to capture output all the time for MapScript, we will
  actually always need the msIO layer.  So USE_MAPIO will always have to 
  be defined.

::

 msIO_resetHandlers();

* Resets msIO handlers to defaults (using stdin, stdout, stderr). 

* Clears buffer data if buffered handlers were installed. 

::

 msIO_installStdoutToBuffer();

* Install handler to send stdout to a buffer, clear buffer. 

::

 msIO_installStdinFromBuffer();

* Install handler to get stdin from a buffer, clear buffer. 

::

 msIO_setStdinBuffer( unsigned char *data, int length );

* Set data for stdin buffer.  

::

 gdBuffer msIO_getStdoutBufferBytes();

* Fetch stdout buffer pointer and length.
* gdBuffer already provides language specific bindings to get byte data.

:: 

  const char *msIO_getStdoutBufferString();

* Fetch stdout buffer as a string.  Appropriate for XML and HTML results for
  instance.

The installed "buffer" handlers will manage their own buffer and concept
of current read/write position.   

My objective is that folks should be able to do something like this in 
Python MapScript. 

::

  mapscript.msIO_installStdoutToBuffer()
  if map.OWSDispatch( req ) == mapscript.MS_SUCCESS:
    result = mapscript.msIO_getStdoutBufferString()
    mapscript.msIO_resetHandlers()

Questions:

1) Should we be "pushing" handlers instead of installing them and losing
track of the previous handler?  Then we could just pop them off.  

2) Should we make the whole msIOContext thing more visible to MapScript?  It 
seems like it would be complicated.

gdBuffer
--------

The msIO_getStdoutBufferBytes() returns a gdBuffer since most language
bindings already have a way of using this as a "array of raw bytes" buffer.
It is normally used for fetched gdImage buffers.  But because the msIO
function returns a gdBuffer referring to an internally memory array not
"owned" by the gdBuffer we need to add a owns_data flag. 

::

 typedef struct {
    unsigned char *data;
    int size;							
    int owns_data;
 } gdBuffer;

Likewise, each of the language bindings needs to be modified to only 
call gdFree() on data if owns_data is true.

This::

 %typemap(out) gdBuffer {
    $result = PyString_FromStringAndSize($1.data, $1.size); 
    gdFree($1.data);
 }

becomes this::

 %typemap(out) gdBuffer {
    $result = PyString_FromStringAndSize($1.data, $1.size); 
    if( $1.owns_data )
	gdFree($1.data);
 }

And similarly for the other bindings. 


Files and objects affected
--------------------------

::

 mapio.c
 mapio.h
 mapscript/mapscript.i
 mapscript/swiginc/owsrequest.i
 mapscript/swiginc/image.i
 mapscript/swiginc/msio.i (new)
 mapscript/python/pymodule.i (gdBuffer)


Backwards compatibility issues
------------------------------

There are no apparent backward compatibility problems with existing
MapScript scripts.


Implementation Issues
---------------------

* the gdBuffer stuff likely ought to be generalized.
* some MapScript languages lack gdBuffer typemaps (ie. perl).
* some performance testing should be done to verify that USE_MAPIO isn't
  going to slow down normal operations significantly.  This is specially
  a concern once the mapio.c statics are actually handled as thread local 
  as each msIO call will need to search for the appropriate thread local 
  context. 
* the msIO "buffer" approach is predicated on streaming output results into
  a memory buffer.  For very large return results this may use an unreasonable 
  amount of memory.  For instance a WFS request with a 250MB response document.
  But such results aren't necessarily reasonable in web services context 
  anyways.
* The set of functions will need to be exposed separately in the PHP bindings.


Test suite
----------

The msautotest/mspython and python unit tests will be extended with at
least rudimentary testing of a few of these services.  

As we have no automated tests for other MapScript languages, no automated
tests will be added, but I will endeavour to prepare simple scripts to test
things.  Currently this has been done for Python and Perl MapScript.


Example
-------

This shows a very simple Python MapScript script that invokes a 
passed in OWSRequest passed via normal cgi means, but adding a text/plain
content type ahead of the regular content type so we can see the results.
The script could easily have done extra manipulation on the URL parameters,
and on the map object.

Example::

 #!/usr/bin/env python

 import sys
 import mapscript

 req = mapscript.OWSRequest()
 req.loadParams()

 mapscript.msIO_installStdoutToBuffer()

 map.OWSDispatch( req )

 print 'Content-type: text/plain'
 print
 print mapscript.msIO_getStdoutBufferString()


Bug ID
------

https://github.com/mapserver/mapserver/issues/1788


Voting history
--------------

+1: FrankW, SteveW. 



Open questions
--------------

None
