|  |  |  |  | 
| File ModulesFile Modules — More about file modules | 
File modules implement loading and saving of SPM files. The structure and operation of file modules will be explained on a simple but complete sample file module. This module imports a sample data format that was construed for this tutorial and does not occur in practice. However, the format resembles many (binary) data formats used in practice, except it contains no device and manufacturer information as they are not essential for the example.
The sample file format our module will load looks as follows, all values are stored in little-endian:
| Position | Type | Name | Description | 
|---|---|---|---|
| 0x00 | char[4] | magic | Magic header "SMPL" | 
| 0x04 | uint16 | xres | Pixels per line | 
| 0x06 | uint16 | yres | Number of lines | 
| 0x08 | float | measure | Size of one pixel [nm] | 
| 0x0c | float | z0 | Value corresponding to 0 in raw data [nm] | 
| 0x10 | float | gain | Conversion factor from raw data to physical values [nm] | 
| 0x14 | uint16[ xres*yres] | data | Raw data values | 
      Furthermore we will assume files of this type get the extension
      .ssd (Simple SPM Data).  The complete module source
      code follows, it will be described in detail below.
    
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | #include <string.h> #include <stdio.h> #include <glib/gstdio.h> #include <libgwyddion/gwymacros.h> #include <libgwyddion/gwymath.h> #include <libprocess/stats.h> #include <libgwymodule/gwymodule-file.h> #include <app/gwymoduleutils-file.h> #include "err.h" #define EXTENSION ".ssd" #define MAGIC "SMPL" #define MAGIC_SIZE (sizeof(MAGIC) - 1) enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 }; typedef struct { guint xres; guint yres; gdouble measure; gdouble z0; gdouble gain; } SimpleFile; static gboolean module_register(void); static gint simple_detect (const GwyFileDetectInfo *fileinfo, gboolean only_name); static GwyContainer* simple_load (const gchar *filename, GwyRunType mode, GError **error); static GwyModuleInfo module_info = { GWY_MODULE_ABI_VERSION, &module_register, N_("Imports simple data files."), "J. Random Hacker <hacker.jr@example.org>", "1.0", "Bit Rot Inc.", "2006", }; GWY_MODULE_QUERY(module_info) static gboolean module_register(void) { gwy_file_func_register("simple", N_("Simple AFM files (.afm)"), (GwyFileDetectFunc)&simple_detect, (GwyFileLoadFunc)&simple_load, NULL, NULL); return TRUE; } static gint simple_detect(const GwyFileDetectInfo *fileinfo, gboolean only_name) { if (only_name) return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0; if (fileinfo->buffer_len > MAGIC_SIZE && memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0) return 100; return 0; } static GwyContainer* simple_load(const gchar *filename, G_GNUC_UNUSED GwyRunType mode, GError **error) { SimpleFile simple; GwyContainer *container = NULL; GwySIUnit *unit; GwyDataField *dfield; guchar *buffer; const guchar *p; GError *err = NULL; gsize size, expected_size; const guint16 *rawdata; gdouble *data; gint i; if (!gwy_file_get_contents(filename, &buffer, &size, &err)) { err_GET_FILE_CONTENTS(error, &err); g_clear_error(&err); return NULL; } if (size <= HEADER_SIZE) { err_TOO_SHORT(error); gwy_file_abandon_contents(buffer, size, NULL); return NULL; } if (memcmp(buffer, MAGIC, MAGIC_SIZE) != 0) { err_FILE_TYPE(error, "Simple"); gwy_file_abandon_contents(buffer, size, NULL); return NULL; } p = buffer + MAGIC_SIZE; simple.xres = gwy_get_guint16_le(&p); simple.yres = gwy_get_guint16_le(&p); simple.measure = gwy_get_gfloat_le(&p); simple.z0 = gwy_get_gfloat_le(&p); simple.gain = gwy_get_gfloat_le(&p); expected_size = 2*simple.xres*simple.yres + HEADER_SIZE; if (size != expected_size) { err_SIZE_MISMATCH(error, expected_size, size); gwy_file_abandon_contents(buffer, size, NULL); return NULL; } simple.measure *= 1e-9; simple.z0 *= 1e-9; simple.gain *= 1e-9; dfield = gwy_data_field_new(simple.xres, simple.yres, simple.xres*simple.measure, simple.yres*simple.measure, FALSE); gwy_convert_raw_data(p, simple.xres*simple.yres, 1, GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN, gwy_data_field_get_data(dfield), simple.gain, simple.offset); unit = gwy_si_unit_new("m"); gwy_data_field_set_si_unit_xy(dfield, unit); g_object_unref(unit); unit = gwy_si_unit_new("m"); gwy_data_field_set_si_unit_z(dfield, unit); g_object_unref(unit); container = gwy_container_new(); gwy_container_set_object_by_name(container, "/0/data", dfield); gwy_container_set_string_by_name(container, "/0/data/title", g_strdup("Topography")); g_object_unref(dfield); gwy_file_channel_import_log_add(container, 0, "simple", filename); gwy_file_abandon_contents(buffer, size, NULL); return container; } | 
Beside standard headers our module includes a special header that provides some common inline functions
| 1 | #include "err.h" | 
      It is located in modules/file for the use by
      in-tree modules.  External modules can copy it (or only the
      actually used functions) to their source code.
    
      Header err.h contains common error reporting
      functions (called err_ERROR_TYPE).
      Its primary purpose it to ensure consistency of error
      messages between file modules and to ease translators' work.
    
Next we define a few convenience symbols: the file extension, the magic header and its size, and header size
| 1 2 3 4 5 6 | #define EXTENSION ".ssd" #define MAGIC "SMPL" #define MAGIC_SIZE (sizeof(MAGIC) - 1) enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 }; | 
We also define a structure representing the imported file. It is a bit superfluous in our module as we do not pass this information around, but it is useful in many modules
| 1 2 3 4 5 6 7 | typedef struct { guint xres; guint yres; gdouble measure; gdouble z0; gdouble gain; } SimpleFile; | 
      The feature registration is similar to data processing functions,
      there are just no menu paths and stock icons, on the other hand there are
      more functions the module can provide: one for each file operation.  Our
      module implements only file detection and import therefore we pass
      NULL for the save and export functions.
    
| 1 2 3 4 5 6 7 8 9 10 11 12 | static gboolean module_register(void) { gwy_file_func_register("simple", N_("Simple AFM files (.afm)"), (GwyFileDetectFunc)&simple_detect, (GwyFileLoadFunc)&simple_load, NULL, NULL); return TRUE; } | 
It is necessary to provide at least one of load, save and export operations, otherwise the file type would be rejected. It is highly recommended to provide a detection function, although it is not required. If no detection function exists for a file type the user has to always explicitly request the type to load and/or save files in this format.
      Two types of detection exist: on load when we have the file
      available, and on save when the file does not exist and we know only
      the requested file name.  They are differentiated by
      only_name argument.  In both cases the detection
      function gets a GwyFileDetectInfo structure, but if
      only_name is FALSE, some fields are unset.
    
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | static gint simple_detect(const GwyFileDetectInfo *fileinfo, gboolean only_name) { if (only_name) return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0; if (fileinfo->buffer_len > MAGIC_SIZE && memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0) return 100; return 0; } | 
The latter case is usually trivial, we just compare the extension with the normal extension of our file type.
      The former case can be more complex.  File name does not tell us much
      about file type: consider just the number of formats using extension
      .afm (not talking about .dat
      which can be essentially anything).  Therefore we use file names only
      as the last resort and base the detection on actual file contents.
    
Many formats have well defined magic headers, precisely for the puropose of easy detection. If out format has one, we are lucky and the detection code can look as simple as in the example. To detect a particular version or type of a file format a more detailed analysis is necessary, but not substantially different from the simple case.
      The most problematic case arises when the format has no detectable magic
      header at all.  Often we can employ the following approach: read the
      header data and check whether it makes sense if interpreted as our
      file format.  For example we read the number of lines and pixels per
      line, calculate expected file size and compare it to the real file size.
      Or we read the physical dimensions and check they are positive (and
      approximately of expected magnitude).  If our detection routine requires
      significant processing or head and
      tail are not sufficient and we have to open
      the file ourselves, it is advisable to perform a quick check that can
      eliminate most files that are not of our type first,
      and then perform the sophisticated test only on the likely candidates.
    
The task of the load function is to read the file and create a GwyContainer containing all importable data. It must not assume anything about the file properties and contents, not even that the file passed the detection because the user can request a file type explicitly, bypassing detection. It has also deal with broken files, checking whether the file is long enough to contain the data it claims to and performing further parameter validation if necessary.
      The file contents can be often read with gwy_file_get_contents() which
      uses mmap() (if available).  The returned memory
      is read-only.  Sometimes destructive header parsing is useful, in such
      a case it is possible to use g_memdup() to duplicate the header,
      keeping the data part mapped read-only.
    
| 1 2 3 4 5 | if (size <= HEADER_SIZE) { err_TOO_SHORT(error); gwy_file_abandon_contents(buffer, size, NULL); return NULL; } | 
The header is parsed using the portable binary buffer reading functions.
| 1 2 3 4 5 6 | p = buffer + MAGIC_SIZE; simple.xres = gwy_get_guint16_le(&p); simple.yres = gwy_get_guint16_le(&p); simple.measure = gwy_get_gfloat_le(&p); simple.z0 = gwy_get_gfloat_le(&p); simple.gain = gwy_get_gfloat_le(&p); | 
      Then we can create the data field and read the data into it.  Since
      Gwyddion always works with values in base
      SI units, we have to convert the raw data values to physical values.
      If the file uses non-base units (which is common in SPM), we have to
      multiply the data by the corresponding factor: in this case with
      10-9 as the physcial quantities are stored
      in nanometers in the file.
      Raw data conversion can be most easily performed with the convenience
      function gwy_convert_raw_data().  Sometimes it may be not sufficient
      so you can resort to the low-level GLib macros such as GUINT16_FROM_LE()
      or Gwyddion get-functions gwy_get_gdouble_be().
    
| 1 2 3 4 5 6 7 8 9 10 11 | simple.measure *= 1e-9; simple.z0 *= 1e-9; simple.gain *= 1e-9; dfield = gwy_data_field_new(simple.xres, simple.yres, simple.xres*simple.measure, simple.yres*simple.measure, FALSE); gwy_convert_raw_data(p, simple.xres*simple.yres, 1, GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN, gwy_data_field_get_data(dfield), simple.gain, simple.offset); | 
We have to set also the lateral and value units of the created data field (by default, the dimensions and values are unitless).
| 1 2 3 4 5 6 7 | unit = gwy_si_unit_new("m"); gwy_data_field_set_si_unit_xy(dfield, unit); g_object_unref(unit); unit = gwy_si_unit_new("m"); gwy_data_field_set_si_unit_z(dfield, unit); g_object_unref(unit); | 
Our sample file format can only store topography channels, therefore the units are always meters. For different types of channels the units have to be set accordingly. Note that
| 1 | unit = gwy_si_unit_new("nA"); | 
is exactly the same as
| 1 | unit = gwy_si_unit_new("A"); | 
      In other words, the prefixes are ignored because GwySIUnit represents
      the unit type and contains no magnitude information.  However, if the
      format stores the units as a string including the prefix we can use
      gwy_si_unit_new_parse() to obtain the corresponding power of 10 (and
      subsequently multiply data field values with the correct multiplier).
    
| 1 | unit = gwy_si_unit_new_parse("nA", &power10); | 
      Finally we create a new GwyContainer, put the data field at
      "/0/data" and set the channel title
      "/0/data/title" (if the file contains explicit
      channel names, they are good candidates for a channel title, otherwise
      we settle for channel type as Topography or Current), and return the
      container.  The application then takes care of adding it to the data
      browser and showing it in a data window.  Calling
      gwy_file_channel_import_log_add() for channel 0 adds a data operation
      log entry, noting that the data originated by import from the
      "simple" format.
    
| 1 2 3 4 5 6 7 8 9 10 11 12 | container = gwy_container_new(); gwy_container_set_object_by_name(container, "/0/data", dfield); gwy_container_set_string_by_name(container, "/0/data/title", g_strdup("Topography")); g_object_unref(dfield); gwy_file_channel_import_log_add(container, 0, "simple", filename); gwy_file_abandon_contents(buffer, size, NULL); return container; | 
      Real file formats often contain more information we would like to
      import.  First, they can contain several data fields (channels or even
      unrelated images).  We store the second data at
      "/1/data" and its title at
      "/1/data/title", the third channel data at
      "/2/data", etc.  The number sequence does not have
      to be contignuous but it is customary to make it so (in any case
      small integers should be used, do not start counting from 12758933).
    
      Auxiliary information about the data as date and time of scanning, scan
      speed, tunneling current, tip oscillation frequency or user-defined
      comments can be stored to metadata.  Metadata is a nested GwyContainer
      at "/0/meta" (for the first channel, the number of
      course corresponds to channel id).  The keys are simply names of the
      values, the values are always strings.  Note both keys and values have
      to be converted to UTF-8.
    
| 1 2 3 4 5 6 7 | meta = gwy_container_new(); gwy_container_set_string_by_name(meta, "Comment", g_strdup(myfile->user_comment)); gwy_container_set_string_by_name(meta, "Tip oscillation frequency", g_strdup_printf("%g Hz", myfile->freq)); gwy_container_set_object_by_name(container, "/0/meta", meta); g_object_unref(meta); | 
Gwyddion runs on many platforms and can be compiled with various compilers. File import is the area most affected by the differences between them, therefore certain care must be taken to make file modules work across all platforms.
        Text files can encode line ends
        differently from the platform Gwyddion is running on.  Therefore
        opening files as text may lead to unexpected results.  One usually
        opens all files as binary and uses gwy_str_next_line() to parse text
        files (or text parts of files) to lines.
      
        The text representation of real numbers is
        locale-dependent.  User's locale is unpredictable and a module must not
        set locale to arbitrary values because it would affect the rest of the
        application.  Therefore one has to use functions as g_ascii_strtod()
        to read real numbers portably.
      
        Many text strings, like comments and remarks, are stored in an
        arbitrary character encoding.
        Typically it is ISO 8895-1 (Latin1) or UTF-16, but sometimes more
        exotic encodings are used.  Since Gwyddion uses UTF-8 for all text data
        (like Gtk+ does) they must be converted, e.g., with g_convert().
      
The canonical bad example is
| 1 2 3 4 5 6 7 | struct { char c; LONG i; wchar_t remark[20]; } header; fread(&header, sizeof(header), 1, filehandle); | 
For start, LONG is unlikely to exist on non-Microsoft platforms. So should we fix it to long int? No, the type size of classic C types (short, long, etc.) is different on different architectures (more precisely ABIs) and the standard prescribes only the lower limit. If we mean `32bit integer' we also have to express it in our code: gint32.
        Likewise, wchar_t size is not well-defined.   If the
        file comes from a Microsoft platform, where wchar_t is
        used a lot, it is probably 16bit, so we write gunichar2 instead
        and use g_utf16_to_utf8() to get UTF-8.
      
        Then we have byte order.  If we are on a platform
        with different native byte order than the file uses, we have to reverse
        the bytes in each multi-byte value.  GLib provides macros such as
        GUINT16_FROM_LE() to simplify this task.
      
        And last but not least, different ABIs prescribe different
        structure padding.  Typically there can be
        from 0 to 7 unused bytes between c and
        i fields.  While most compilers provide
        some packing #pragmas or other means to
        specify desired structure padding, the portability of these
        constructs is questionable at best.
      
The answer to (almost) all the problems above is: Don't do that then. Avoid direct reading of structures, read the data into a flat byte buffer instead and use the file module utils functions to obtain individual values.