sil2100//vx developer log

Welcome to my personal web page


Flatconf 1.5 concept?

The Flatconf project has been part of my interests since long - from the very first moment I started cooperating the ASN Labs team. Not a very well known project, from what I know it was only used in the old Lintrack distribution. It's more of an interesting experimental concept than an innovative solution. Flatconf is an attempt of creating an universal configuration system based on the idea of 'flat files' and the usage of the file-system as a natural database. Currently two versions of the specification exist, with the newer one (2.0 draft) using a new concept for holding variable meta-data - not entirely a good one though. But, just now, I thought about the original idea and came up with some thoughts for modifying it, making it little bit more feasible. Flatconf 1.5 anyone?

2011-07-06 12:13

The original concept of Flatconf 1.0 had uber-flatness in mind. To understand the motivation, first we need to define what a flat file means in our context. A flat file is a file which contents are not structured in any additional way besides holding data. This means, that to get the actual 'data' from a flat file means just reading its contents - because its contents are plain data with no unnecessary structural meta-data. A structured (not flat) file would be any file which would have to be parsed first to retrieve the data of interest.

Flatconf used flat files for holding configuration variables, i.e. a separate file for every variable. Structurization of data is achieved by using the file-system directory hierarchy. Each variable can have meta-data associated with it - such as information about variable type, variable user-readable description and many more. The main problem and difference between versions 1.0 and 2.0 is the way these meta-data are stored.

The 1.0 approach had the problem of bloating the whole data directory tree with unnecessary files and directories at every hierarchy level. The 2.0 approach, well, wasn't flat anymore, and since it used a newly designed meta language - created many, many problems and had many drawbacks. It removed the ease of variable manipulation - forcing the system to parse meta-data first, adding complexity.

But what if we would still leave the 1.0 flat concept, although slightly modified? What if we just moved all the meta-data from the data tree to a separate directory like in 2.0, but left them as flat files? This requires the overall specification to be changed, since now we might have a bit more freedom.
Best explaining it with examples of Flatconf configuration trees.

data/                # Main fc data directory
 |- hostname         # - \
 |- welcome_text     # - Text variables
 \- net/             # A directory variable
     |- ip
     |- gw
     \- ifaces/      # A list variable, i.e. a directory with user element-addition possibilities
         |- eth0/    # eth0 interface element added by user
         \- wlan0/   # wlan0 interface element added by user
                     # (both are directories holding more variables inside, like MAC address etc.)

metadata/               # Main fc directory for holding respective meta-data
 |- hostname/           # Every variable has a directory named the same way holding its meta-data
 |   |- .type           # Every meta-data is a separate file, starting with a '.'
 |   |- .descr          # e.g. this is the variable human-readable description
 |   \- .onapply        # And this is a script/application that is fired on variable modification
 |- welcome_text/
 |   \- (...)
 \- net/
     |- .type           # e.g. directories have "dir" in their .type contents
     |- .descr-en
     |- .descr-pl       # Descriptions can be locale-specific
     |- ip/
     |- gw/
     \- ifaces/
         |- .type
         |- .skel/      # The skeleton directory, holding the base meta-data for every new list element
         |   |- .type   # Like for instance a .type data with contents "dir"
         |   \- (...)
         |- eth0/       # Every new list element has a named copy of .skel created
         |   |- .type   # Same contents as in .skel/
         |   \- (...)
         \- wlan0/

This example can be better understood if one already knows the basics behind the Flatconf concept.

In this approach, fetching the description string meta-data for the net/ip variable is simple as issuing cat $FC_METATREE/net/ip/.descr in bash. The same for fetching the variable's data: cat $FC_DATATREE/net/ip.
Of course, in this approach, the meta-tree is not read-only anymore, as addition/deletion of new elements to the list involves creating/removing directories from the list's meta-data directory. But for this concept it is of not much concern.

Why meta-data variables start with a '.' (dot) character? To remove ambiguity. Because of this small modification, we can easily create directories holding new element meta-data named the same way as the element being added, without having to worry that someone would add a descr list element, bringing confusion to the system (confusion like: "is this the descr meta-data variable, or a list element called descr?")
As for list ordering and visibility - we could add a .listorder trivially structured meta-data variable listing the element order or even list element visibility.
Well, all this is just an idea.

As stated before, Flatconf is an experiment. Using the file-system as a natural database for structuring and holding variables is an interesting idea, similar to the GNU/Linux concept of 'everything is a file'. Actually, Flatconf is very similar - in visualization - to the /proc/sys sysctl interface. But is it efficient? When writing bash scripts, it might indeed be much easier just browsing the file-system and reading file contents instead of parsing contents by hand. But when writing C/C++ code - the costs might be higher. One of these days I will do some benchmarks to test performance issues.

I noticed that many of my posts are about less-known, niche topics. This does not mean the unfamiliar cannot be interesting. There is always much to learn from the non-mainstream!