Copyright © 2012 Free Software Foundation, Inc.

This is Edition 1 of Writing Extensions for Gawk: A Temporary Manual, for the 4.1.0 (or later) version of the GNU implementation of AWK.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with the Invariant Sections being ``GNU General Public License'', the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below).  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

 a. ``A GNU Manual''

 b. ``You have the freedom to copy and modify this GNU manual.  Buying copies from the FSF supports it in developing GNU and promoting software freedom.''

Writing Extensions for gawk
***************************

This chapter describes how to extend gawk using code written in C or C++.  If you don't know anything about C programming, you can safely skip this chapter, although you may wish to review the documentation on the extensions that come with gawk (see Extension Samples).

Introduction
============

An extension (sometimes called a plug-in) is a piece of external code that gawk can load at run-time to provide additional functionality, over and above the built-in capabilities described in the rest of this book.

Extensions are useful because they allow you (of course) to extend gawk's functionality. For example, they can provide access to system calls (such as chdir() to change directory) and to other C library routines that could be of use.  As with most software, ``the sky is the limit;'' if you can imagine something that you might want to do and can write in C or C++, you can write an extension to do it!

Extensions are written in C or C++, using the Application Programming Interface (API) defined for this purpose by the gawk developers.  The rest of this chapter explains the design decisions behind the API, the facilities it provides and how to use them, and presents a small sample extension.  In addition, it documents the sample extensions included in the gawk distribution.

Extension API Design
====================

The first version of extensions for gawk was developed in the mid-1990s and released with gawk 3.1 in the late 1990s.  The basic mechanisms and design remained unchanged for close to 15 years, until 2012.

The old extension mechanism used data types and functions from gawk itself, with a ``clever hack'' to install extension functions.

gawk included some sample extensions, of which a few were really useful.  However, it was clear from the outset that the extension mechanism was bolted onto the side and was not really thought out.

Problems With The Old Mechanism
-------------------------------

The old extension mechanism had several problems:

  * It depended heavily upon gawk internals. Any time the NODE structure changed, an extension would have to be recompiled. Furthermore, to really write extensions required understanding something about gawk's internal functions.  There was some documentation in this book, but it was quite minimal.

  * Being able to call into gawk from an extension required linker facilities that are common on Unix-derived systems but that did not work on Windows systems; users wanting extensions on Windows had to statically link them into gawk, even though Windows supports dynamic loading of shared objects.

  * The API would change occasionally as gawk changed; no compatibility between versions was ever offered or planned for.

Despite the drawbacks, the xgawk project developers forked gawk and developed several significant extensions. They also enhanced gawk's facilities relating to file inclusion and shared object access.

A new API was desired for a long time, but only in 2012 did the gawk maintainer and the xgawk developers finally start working on it together. More information about the xgawk project is provided in gawkextlib.

Goals For A New Mechansim
-------------------------

Some goals for the new API were:

  * The API should be independent of gawk internals.  Changes in gawk internals should not be visible to the writer of an extension function.

  * The API should provide binary compatibility across gawk releases as long as the API itself does not change.

  * The API should enable extensions written in C to have roughly the same ``appearance'' as awk functions, meaning:

       - The ability to access function parameters.

       - The ability to turn an undefined parameter into an array (call by reference).

       - The ability to create, access and update global variables.

       - It should provide easy access to all the elements of an array at once (``array flattening'') in order to loop over all the element in an easy fashion for C code.

  * The ability to create arrays (including gawk's true multi-dimensional arrays).

  * The API should use only features in ISO C 90, so that extensions can be written using the widest range of C and C++ compilers. The header should include the appropriate #ifdef __cplusplus and extern "C" magic so that a C++ compiler could be used.  (If using the C++, the runtime system has to be smart enough to call any constructors and destructors, as gawk is a C program.)

  * The API mechanism should not require access to gawk's symbols (The symbols are the variables and functions defined inside gawk.  Access to these symbols by code external to gawk loaded dynamically at run-time is problematic on Windows.) by the compile-time or dynamic linker, in order to enable creation of extensions that will also work on Windows.

During development, it became clear that there were other features that should be available to extensions, which were also subsequently provided:

  * Extensions should have the ability to hook into gawk's I/O redirection mechanism.  In particular, the xgawk developers provided a so-called ``open hook'' to take over reading records.  During the development, this was generalized to allow extensions to hook into input processing, output processing, and two-way I/O.

  * An extension should be able to provide a ``call back'' function to perform clean up actions when gawk exits.

strongFIXME: Review the header for other things to list here.

Other Design Decisions
----------------------

As an ``arbitrary'' design decision, extensions read the values of built-in variables and arrays (such as ARGV, FS), but cannot change them, with the exception of PROCINFO.

The reason for this is to prevent an extension function from affecting the flow of an awk program outside its control.  While a real awk function can do what it likes, that is at the discretion of the programmer.  An extension function should provide a service or make a C API available for use within awk, and not mess with FS or ARGC and ARGV.

In addition, it becomes easy to start down a slippery slope. How much access to gawk facilities do extensions need? Do they need getline?  What about calling gsub() or compiling regular expressions?  What about calling into awk functions? (That would be messy.)

In order to avoid these issues, the gawk developers chose to start with the simplest, most basic features that are still truly useful.

Another decision is that although gawk provides nice things like MPFR, and arrays indexed internally by integers, we are not bringing these features out to the API in order to keep things simple and close to traditional awk semantics.  (In fact, arrays indexed internally by integers are so transparent that they aren't even documented!)

At A High Level How It Works
----------------------------

The requirement to avoid access to gawk's symbols is, at first glance, a difficult one to meet.

One design, apparently used by Perl and Ruby and maybe others, would be to make the mainline gawk code into a library, with the gawk program a small C main() function linked against the library.

This seemed like the tail wagging the dog, complicating build and installation and making a simple copy of the gawk executable from one system to another (or one place to another on the same system!) into a chancy operation.

Pat Rankin suggested the solution that was adopted. Communication between gawk and an extension is two way.  First, when an extension is loaded, it is passed a pointer to a struct whose fields are function pointers.

The extension can call functions inside gawk through these function pointers, at runtime, without needing (link time) access to gawk's symbols.  One of these function pointers is to a function for ``registering'' new built-in functions.

In the other direction, the extension registers its new functions with gawk by passing function pointers to the functions that provide the new feature (do_chdir(), for example).  gawk associates the function pointer with a name and can then call it, using a defined calling convention. The do_XXX() function, in turn, then uses the function pointers in the API struct to do its work.

Convenience macros in the gawkapi.h header file make calling through the function pointers look like regular function calls so that extension code is quite readable and understandable.

Although all of this sounds medium complicated, the result is that extension code is quite clean and straightforward. This can be seen in the sample extensions filefuncs.c and also the testext.c code for testing the APIs.

Some other bits and pieces:

  * The API provides access to gawk's do_XXX values, reflecting command line options, like do_lint, do_profiling and so on (see Extension API Variables). These are informational: an extension cannot affect these inside gawk.  In addtion, attempting to assign to them produces a compile-time error.

  * The API also provides major and minor version numbers, so that an extension can check if the gawk it is loaded with supports the facilties it was compiled with.  (Version mismatches ``shouldn't'' happen, but we all know how that goes.)

  * An extension may register a version string with gawk; this allows gawk to dump extension version information when invoked with the --version option.

Room For Future Growth
----------------------

The API also provides room for future growth, in two ways.

An ``extension id'' is passed into the extension when its loaded. This extension id is then passed back to gawk with each function call.  This allows gawk to identify the extension calling it, should it need to know.

A ``name space'' is passed into gawk when an extension is registered.  This allows for some future mechanism for grouping extension functions and possibly avoiding name conflicts.

Of course, as of this writing, no decisions have been made with respect to any of the above.

API Versioning
--------------

The API provides both a ``major'' and a ``minor'' version number. The API versions are available at compile time as constants:

GAWK_API_MAJOR_VERSION
     The major version of the API.

GAWK_API_MINOR_VERSION
     The minor version of the API.

The minor version increases when new functions are added to the API. Such new functions are always added to the end of the API struct.

The major version increases (and the minor version is reset to zero) if any of the data types change size or member order, or if any of the existing functions change signature.

It could happen that an extension may be compiled against one version of the API but loaded by a version of gawk using a different version. For this reason, the major and minor API versions of the running gawk are included in the API struct as read-only constant integers:

api->major_version
     The major version of the running gawk.

api->minor_version
     The minor version of the running gawk.

It is up to the extension to decide if there are API incompatibilities. Typically a check like this is enough:

     if (api->major_version != GAWK_API_MAJOR_VERSION
         || api->minor_version < GAWK_API_MINOR_VERSION) {
             fprintf(stderr, "foo_extension: version mismatch with gawk!\n");
             fprintf(stderr, "\tmy version (%d, %d), gawk version (%d, %d)\n",
                     GAWK_API_MAJOR_VERSION, GAWK_API_MINOR_VERSION,
                     api->major_version, api->minor_version);
             exit(1);
     }

Such code is included in the boilerplate dl_load_func macro provided in gawkapi.h.

API Description
===============

As the API has evolved, it has settled into a pattern where query routines return an awk_bool_t, with ``true'' meaning success and ``false'' not, but a false return still fills in the actual type.

Data Types
----------

Functions
---------

Access to facilities within gawk are made available by calling through function pointers passed into your extension.

API function pointers are provided for the following kinds of operations:

  * Accessing parameters, including converting an undefined paramater into array

  * Printing fatal, warning, and lint warning messages

  * Registering input parsers, output wrappers, and two-way processors

  * Updating ERRNO, or unsetting it

  * Registering an extension function

  * Registering exit handler functions to be called when gawk exits

  * Accessing and creating global variables

  * Symbol table access: retreiving a global variable, creating one, or changing one. This also includes the ability to create a scalar variable that will be constant within awk code.

  * Manipulating arrays

       - Retrieving, adding, deleting, and modifying elements

       - Getting the count of elements in an array

       - Creating a new array

       - Clearing an array

       - Flattening an array for easy C style looping over an array

  * Creating and releasing cached values; this provides an efficient way to use values for multiple variables and can be a big performance win.

  * Registering an informational version string.

While you may call through these function pointers directly, the interface is not so pretty. To make extension code look more like regular code, the gawkapi.h header file defines a number of macros which you should use in your code. This section presents the macros as if they were functions.

Points about using the API:

All pointers filled in by gawk are to memory managed by gawk and should be treated by the extension as read-only.  Memory for all strings passed into gawk from the extension must come from malloc() and is managed by gawk from then on.

The API defines several simple structs that map values as seen from awk.  A value can be a double, a string, or an array (as in multidimensional arrays, or when creating a new array). Strings maintain both pointer and length since embedded NUL characters are allowed.

By intent, strings are maintained using the current multibyte encoding (as defined by LC_XXX environment variables) and not using wide characters.  This matches how gawk stores strings internally and also how characters are likely to be input and output from files.

When retrieving a value (such as a parameter or that of a global variable or array element), the extension requests a specific type (number, string, array, or ``undefined'').  When the request is undefined, the returned value will have the real underlying type.

However, if the request and actual type don't match, the access function returns ``false'' and fills in the type of the actual value that is there, so that the extension can, e.g., print an error message (``scalar passed where array expected'').

Customized Input Parsers
........................

By default, gawk reads text files as its input. It uses the value of RS to find the end of the record, and then uses FS (or FIELDWIDTHS) to split it into fields.  Additionally, it sets the value of RT.  (FIXME: pxrefs as needed.)

If you want, you can provide your own, custom, input parser.  An input parser's job is to return a record to the gawk record processing code, along with indicators for the value and length of the data to be used for RT, if any.

To provide an input parser, you must provide two functions (where XXX is a prefix name for your extension):

int XXX_can_take_file(const IOBUF_PUBLIC *iobuf)
     This function examines the information available in iobuf (which we discuss shortly).  Based on the information there, it decides if the input parser should be used for this file. If so, it should return true (non-zero). Otherwise, it should return false (zero).

int XXX_take_control_of(IOBUF_PUBLIC *iobuf)
     When gawk decides to hand control of the file over to the input parser, it calls this function.  This function in turn must fill in certain fields in the IOBUF_PUBLIC structure, and ensure that certain conditions are true.  It should then return true. If an error of some kind occurs, it should not fill in any fields, and should return false; then gawk will not use the input parser. The details are presented shortly.

Your extension should package these functions inside an awk_input_parser_t, which looks like this (from gawkapi.h):

     typedef struct input_parser {
         const char *name;   /* name of parser */
         int (*can_take_file)(const IOBUF_PUBLIC *iobuf);
         int (*take_control_of)(IOBUF_PUBLIC *iobuf);
         struct input_parser *awk_const next;    /* for use by gawk */
     } awk_input_parser_t;

The steps are as follows:

 1. Create a static awk_input_parser_t variable and initialize it appropriately.

 2. When your extension is loaded, register your input parser with gawk using the register_input_parser() API.

An IOBUF_PUBLIC looks like this:

     typedef struct iobuf_public {
         const char *name;       /* filename */
         int fd;                 /* file descriptor */
     #define INVALID_HANDLE (-1)
         void *opaque;           /* private data for input parsers */
         /*
          * The get_record function is called to read the next record of data.
          * It should return the length of the input record (or EOF), and
          * it should set *out to point to the contents of $0.  Note that
          * gawk will make a copy of the record in *out, so the parser is
          * responsible for managing its own memory buffer.  If an error
          * occurs, the function should return EOF and set *errcode
          * to a non-zero value.  In that case, if *errcode does not equal
          * -1, gawk will automatically update the ERRNO variable based on
          * the value of *errcode (e.g. setting *errcode = errno should do
          * the right thing).  It is guaranteed that errcode is a valid
          * pointer, so there is no need to test for a NULL value.  The
          * caller sets *errcode to 0, so there is no need to set it unless
          * an error occurs.  The rt_start and rt_len arguments should be
          * used to return RT to gawk.  Gawk will make its own copy of RT,
          * so the parser is responsible for managing this memory.  If EOF is
          * not returned, the parser must set *rt_len (and *rt_start if *rt_len
          * is non-zero).
          */
         int (*get_record)(char **out, struct iobuf_public *, int *errcode,
                 char **rt_start, size_t *rt_len);
         /*
          * The close_func is called to allow the parser to free private data.
          * Gawk itself will close the fd unless close_func sets it to -1.
          */
         void (*close_func)(struct iobuf_public *);

         /* put last, for alignment. bleah */
         struct stat sbuf;       /* stat buf */
     } IOBUF_PUBLIC;

The fields can be divided into two categories: those for use (initially, at least) by XXX_can_take_file(), and those for use by XXX_take_control_of().  The first group of fields and their uses are as follows:

const char *name;
     The name of the file.

int fd;
     A file descriptor for the file.  If gawk was able to open the file, then it will not be equal to INVALID_HANDLE. Otherwise, it will.

struct stat sbuf;
     If file descriptor is valid, then gawk will have filled in this structure with a call to the fstat() system call.

The XXX_can_take_file() function should examine these fields and decide if the input parser should be used for the file. The decision can be made based upon gawk state (the value of a variable defined previously by the extension and set by awk code), the name of the file, whether or not the file descriptor is valid, the information in the struct stat, or any combination of the above.

Once XXX_can_take_file() has returned true, and gawk has decided to use your input parser, it will call XXX_take_control_of().  That function then fills in at least the get_record field of the IOBUF_PUBLIC.  It must also ensure that fd is not set to INVALID_HANDLE.  All of the fields that may be filled by XXX_take_control_of() are as follows:

void *opaque;
     This is used to hold any state information needed by the input parser for this file.  It is ``opaque'' to gawk.  The input parser is not required to use this pointer.

int (*get_record)(char **out, struct iobuf_public *, int *errcode,
char **rt_start, size_t *rt_len);
     This is a function pointer that should be set to point to the function that creates the input records. Said function is the core of the input parser.  Its behavior is described below.

void (*close_func)(struct iobuf_public *);
     This is a function pointer that should be set to point to the function that does the ``tear down.'' It should release any resources allocated by XXX_take_control_of().  It may also close the file. If it does so, it shold set the fd field to INVALID_HANDLE.

     Having a ``tear down'' function is optional. If your input parser does not need it, do not set this field.  In that case, gawk will close the regular close() system call on the file descriptor, so it should be valid.

The XXX_get_record() function does the work of creating input records.  The parameters are as follows:

char **out
     This is a pointer to a char * variable which is set to point to the record.  gawk will make its own copy of the data, so the extension must manage this storage.

struct iobuf_public *iobuf
     This is the IOBUF_PUBLIC for the file.  The fields should be used for reading data (fd) and for managing private state (opaque), if any.

int *errcode
     If an error occurs, *errcode should be set to an appropriate code from <errno.h>.

char **rt_start
size_t *rt_len
     If the concept of a ``record terminator'' makes sense, then *rt_start should be set to point to the data to be used for RT, and *rt_len should be set to the length of the data. Otherwise, *rt_len should be set to zero.

The return value is the length of the buffer pointed to by *out, or EOF if end-of-file was reached or an error occurred.

gawk ships with a sample extension (see Extension Sample Readdir) that reads directories, returning records for each entry in the directory.  You may wish to use that code as a guide for writing your own input parser.

When writing an input parser, you should think about (and document) how it is expected to interact with awk code.  You may want it to always be called, and take effect as appropriate (as the readdir extension does).  Or you may want it to take effect based upon the value of an awk variable, as the XML extension from the gawkextlib project does (see gawkextlib). In the latter case, code in a BEGINFILE section (FIXME: pxref) can look at FILENAME and ERRNO to decide whether or not to activate an input parser.

Customized Output Wrappers
..........................

Customized Two-way Processors
.............................

External Variables
------------------

The API provides access to several variables that describe whether the corresponding command-line options were enabled when gawk was invoked.  The variables are:

do_lint
     This variable will be true if the --lint option was passed (FIXME: pxref).

do_traditional
     This variable will be true if the --traditional option was passed.

do_profile
     This variable will be true if the --profile option was passed.

do_sandbox
     This variable will be true if the --sandbox option was passed.

do_debug
     This variable will be true if the --debug option was passed.

do_mpfr
     This variable will be true if the --bignum option was passed.

The value of do_lint can change if awk code modifies the LINT built-in variable (FIXME: pxref). The others should not change during execution.

Boilerplate Code
----------------

Example: Some File Functions
============================

Sample Extensions
=================

File Related Functions
----------------------

Interface To fnmatch()
----------------------

Interface to fork(), wait() and waitpid()
-----------------------------------------

Character and Numeric values: ord() and chr()
---------------------------------------------

Reading Directories
-------------------

Reading An Entire File
----------------------

Dumping and Restoring An Array
------------------------------

API Tests
---------

Time Functions
--------------

These functions  can be used by either invoking gawk with a command-line argument of -l time or by inserting @load "time" in your script.

gettimeofday()
     This function returns the time that has elapsed since 1970-01-01 UTC as a floating point value.  It should have sub-second precision, but the actual precision will vary based on the platform.  If the time is unavailable on this platform, it returns -1 and sets ERRNO. If the standard C gettimeofday() system call is available on this platform, then it simply returns the value.  Otherwise, if on Windows, it tries to use GetSystemTimeAsFileTime().

sleep(SECONDS)
     This function attempts to sleep for SECONDS seconds. Note that SECONDS may be a floating-point (non-integral) value. If SECONDS is negative, or the attempt to sleep fails, then it returns -1 and sets ERRNO. Otherwise, the function should return 0 after sleeping for the indicated amount of time.  Implementation details: depending on platform availability, it tries to use nanosleep() or select() to implement the delay.

The gawkextlib Project
======================

The gawkextlib (http://sourceforge.net/projects/gawkextlib/) project provides a number of gawk extensions, including one for processing XML files.  This is the evolution of the original xgawk (XML gawk) project.

As of this writing, there are four extensions:

  * XML parser extension, using the Expat (http://expat.sourceforge.net) XML parsing library

  * Postgres SQL extension

  * GD graphics library extension

  * MPFR library extension. This provides access to a number of MPFR functions which gawk's native MPFR support does not.

The time extension described earlier (see Extension Sample Time) was originally from this project but has been moved in to the main gawk distribution.