Introducing sisula

What in the world is sisula? Sisula stands for ‘simple substitution language’ and it is the key to supporting other databases in our modeling tool. Up until now, the tool was using XSLT to transform XML to SQL. But as it turns out, there are very few people who are fluent in both languages. In that respect, using XSLT was a bad design choice. It also becomes verbose, to say the least, and the code looks quite far from the end result.

So, what we are doing right now is moving towards a template language, in which you write SQL code with some variable names in it. It looks very much like the end result, meaning you only need to have very basic programming skills on top of your SQL expertise in order to write templates. We believe that we will get quicker adoption for other databases using this.

Sisulas are actually a family of languages, and we created a flavor suitable for producing SQL, in which the template code is substituted to JavaScript and evaluated to get the end result. You could say that a sisula is a language in which you introduce a simplification or notation that gets substituted to another language using regular expression replacements. Below is what sisula code looks like, with the template parts delimited by tilde signs.

We also wanted a simple way to access the different parts of a model in sisula. Instead of using the XML, and access it using either XPath notation or through DOM traversal, we decided to expose a JSON-compatible object. That way you can use regular object notation to access the parts, which is more familiar and natural. Below is the JSON description of the schema object used in the template above.

Finally, keeping all code together in a single file, as it was previously, turns out to be somewhat limiting as well. By instead having small templates that encapsulate specific pieces of SQL, it makes it much easier for anyone to insert custom templates in between existing ones, or to leave out certain templates altogether. The example above is contained in its own file: CreateKnots.js. Templates are referenced from a directive, that keeps the process together and lists all the template files that should be processed and in which order.

Our Sisulator (some 100 lines of JavaScript code) does all the work, first converting the XML representation of the model to a JSON object, reads the directive, processes each template file in order, substitutes all template code to JavaScript, evaluates the code, and returns the resulting SQL code. Once done, the example given above becomes the SQL below and similarly for all the other knots in your model.

While the Sisulator is in place, we still haven’t ported the old SQL generation to sisula. We expect to have this done before august, and at the same time add concurrent-temporal generation to the tool.

Ensemble Modeling

Hans Hultgren came up with a great name, Ensemble Modeling, for a family of modeling technologies that all embrace the decomposition of an entity into its constituents. Exactly how this separation is made depends on the flavor you are working with, but common to all of them is that you make assumptions on which parts are mutable and which are immutable. The least restrictive is Anchor Modeling, in which you only assume that the identity of an entity is immutable, represented by a surrogate key in a database implementation. Natural keys are mutable in Anchor Modeling. The techniques also differ in terms of grouping and how constraints are maintained. In Anchor Modeling there is no grouping, and constraints are put in place that ensure temporal entity integrity and temporal referential integrity in the database, even for concurrent-temporal models.

At the recent conference in the Netherlands we discussed such differences with Hans in one of the sessions, and the two debaters Hans and Lars were kind enough to put together a short paper on what was said and things they did not have time to say. It is available through Hans’ blog, by clicking here. Hans also made a recent blog post in which he further discusses Ensemble Modeling styles, available by clicking here. Both are useful reading for anyone interested in the commonalities and differences between these techniques.

As more techniques will fall into this category of modeling, we believe that it is good to have a name for the family, similar to how star and snowflake fit under dimensional modeling.

Next Generation DWH Modeling

Next Generation DWH Modeling

We’re back from the event yesterday in the Netherlands, where we were happy to see close to 300 participants and present concurrent-temporal Anchor Modeling for the first time. We were also happy to see Hans Hultgren gather various techniques that separate mutable from immutable content under the same name, Ensemble Modeling. We also heard Ivo Clemens from m–wise challenge our claims of how Anchor Modeling is beneficial compared to other techniques, and coming to the conclusion that almost all of them have been fulfilled in their business, from the experience of building ten data warehouses using it. One of the more memorable quotes was how a client of theirs had their data warehouse up and running in a couple of weeks!

Martijn Evers saw us through a Clash of the Supermodels and Ronald Kunenborg through the history of modeling, as well as presenting differences and similarities between Data Vault and Anchor Modeling in two sessions. We finished off the day battling it out with Hans Hultgren concerning differences between Data Vault and Anchor Modeling. All in all it was a great conference, with lots of fun, and in the spirit of cooperation rather than competition. We’d like to thank Simone Molenaar, BI-Podium, and everyone at Visser & Van Baars for their monumental effort pulling this together.

IMG_1618_2048 IMG_1619_2048 IMG_1621_2048 IMG_1622_2048 IMG_1623_2048 IMG_1617_2048 IMG_1632_2048 IMG_1633_2048 IMG_1627_2048 IMG_1628_2048 IMG_1630_2048 Next Generation DWH Modeling

Photos courtesy of Rob Vonk, who was thoughtful enough to bring a camera.

Concurrency and Reliability

At the Next Generation DWH Modeling event tomorrow, we are proud to announce concurrent-temporal Anchor Modeling, which adds support for multi-temporal modeling, information reliability, and enhanced time traveling. The presentation, with the subtitle “What if you could travel through time and become someone else?” is available by clicking this link: Back to the Moment.

We finished the research during the autumn of last year and have since been working on producing a scientific paper and implementing the code generation in our modeling tool. Neither is yet finished, but we have enough material, and a couple of actual (hand-made) implementations, to go public. We expect to have code generation finished after the summer and the paper to follow after that. There is an example showing inflation estimates using concurrent-temporal modeling available by clicking here.

The Simple Bitemporal Test made using concurrent-temporal Anchor Modeling can be found by clicking here. Please note that this was an early experiment, and that we have changed the way the restatement checking works, as well as added insert, update, and delete triggers since. It does, however, convey the general idea of concurrency and reliability.

Graphical Profile

We have made a logotype and some smaller icons that you can use to show that you support or use Anchor Modeling. Below are low-res PNG and vector SVG formats of the different logotypes and icons:

Anchor

Anchor

Anchor_icon_with_anchorAnchor_icon_with_anchorAnchor_icon_without_anchorAnchor_icon_without_anchor

Feel free to use them. The color is #c80514 or RGB(200, 5, 20), and the two squares symbolise anchors. The font in the trademark symbol is Arial with reduced character spacing, and ANCHOR is written using Magmawave by Yanko Tsvetkov available under a Creative Commons License:

MagmaWave Caps Font (Yanko Tsvetkov) / CC BY-NC 3.0

DMZ Europe and America

We are proud to be presenting Anchor Modeling at the Data Modeling Zone in Europe (Hannover, Germany) on September 23rd-24th and in America (Baltimore, Maryland) on October 8th-10th. Even better is that if you use the promotional code: LarsDMZPromo when you register for America or: LarsDMZEuropePromo for Europe you will recieve a $100 discount of the registration price! We hope to see you there.

Version 0.96 is now live

We’ve just pushed version 0.96 from test to release, making it available by clicking here or on the “Online Modeler” menu item. Please see the three previous posts for details of what has changed since 0.95 (which is still available online here). As well as the listed changes, the following has been done:

  • Fixed a bug in the column name for attributes in the insert trigger.
  • Fixed bug in update trigger for knotted historized attributes.
  • Fixed bug where the navigation miniature was updated at every frame of a dragging operation.
  • Fixed a bug where the modeler could crash after a delete operation, trying to write out a non existent node name.
  • Fixed a bug in the update trigger for the ChangedAt column.
  • Fixed a bug where the miniature could get stuck in a resize loop causing the browser to hang.
  • Changed the edge stiffness algorithm (back) to a more physically accurate one.

This also means we are beginning development of 0.97, of which a test version soon will be made available. Trust me that we have some very interesting features coming in the next version!

Space partitioning

Graph layout with space partitioning
Showing the debug information for space partitions in the graph layout.

We have added space partitioning to the graph layout engine in the modeling tool (rev. 411 now live in test). This will improve performance for very large graphs. The idea is to first divide the space containing the graph into a number of sections called partitions, then calculate the “center of mass” for each partition with respect to the contained nodes, and then calculate the collective repelling force in each partition by the other partitions. Rather than having to do an exponentially growing iteration over nodes and other nodes, we can now iterate only over nodes in the same partition, calculate their respective forces with an addition of the already calculated force in the partition, and move to the next partition.

Looking at recent research in graph layout, an octree is used for the partitioning. However, constructing an octree was not fast enough, so we decided upon another method. The partitioning is made by dividing the node coordinates by the partition size and converting the result to an integer. A hashtable is used to contain the partitions with a pointer to an array of the contained nodes.

This works well, except that since we are continuously moving nodes around, we may get into a situation where nodes are “jumping” between two different partitions, and the layout never reaches equilibrium. We were able to remove such jittering by making the partitions fuzzy, with respect to their size. Every iteration of the layout, the size of a partition will be somewhat different, resulting in the nodes always settling down in a single partition.

In the screenshot above you can see the different partitions in grey (switch on debugging in the test version to see for yourself), and a grey small square in each partition indicating its center of mass.

New features in test

We have updated the test version of the modeling tool to version 0.96 revision 407:

https://www.anchormodeling.com/modeler/test

Updates in revision 407 (and earlier):

  • Fixed bug with restatement control rendered useless in bitemporal models.
  • Added restatement control for ties.
  • Fixed bug where setInterval was being called too many times in older browsers.
  • Replaced setInterval code with setTimeout in older browsers.
  • Sped up rendering even further by pushing run once-if-statements to init-functions.
  • Added insert triggers on the latest views for ties, with idempotency control.
  • Fixed all insert triggers such that when knots are involved, either the ID or the value column from the knot can be used, whereas previously the value column had to be specified.
  • Added delete keyboard shortcut ‘d’.
  • Added a (long overdue) consistency check for ties, such that roles cannot become “deep-equal”, according to the XML Schema for anchor models.
  • Roles in a tie can now have the same name, as long as they do not refer to the same anchor.
  • Removed triggers from anchors that have no attributes (to fix broken code generation).
  • Temporalization metadata is now recognized when models are loaded.
  • Fixed bug that made input fields repeat more than once when selecting bitemporal output.
  • Improvements of how knots are handled in the triggers.
  • The naming convention now allows upper and lower case Latin-1 characters.
  • The naming convention is changed in order to support multiple attributes on the same anchor referencing the same knot. In order to be consistent, the name of the identity in the attribute was changed accordingly.
    This change is NOT backwards compatible!
  • Edit (revision 409 released on Oct 30):
    Added an option in the “Defaults” menu to change back to the “Original” naming convention. This quickly became a public request!

Use this with some caution as some changes are not backwards compatible. Should you decide to use this version you cannot go back to a previous version. You cannot mix the output from this version with an existing implementation either, since the naming convention will differ. However, it should be quite easy to migrate to a new version if you have a new and an old implementation side by side.

Modeler 0.96 available for testing

We have set up a development snapshot of the modeler (v0.96) in order to make it available for testing. It can be reached by clicking the link below:

https://www.anchormodeling.com/modeler/test

This version currently includes improvements to the graphical user interface, an abundance of descriptive tooltips, new and more perfomant rendering in the very latest browsers, and idempotency control.

Idempotency controls whether the insert and update triggers should ignore and discard any attempts to insert or update a value that is identical to the currently stored value (in changing time). Say for example that the attribute “Surname” has the value “Rönnbäck” from “2004-06-19” and that it has been marked as idempotent. Then any attempt to insert the value “Rönnbäck” with a later changing time will be silently ignored.

As such, idempotency can be of great use in systems that receive many repeated values where it also can be ensured that information arrive in changing time order. In a situation where information may arrive out of changing time order, idempotency should be avoided, since there is no way of knowing what values may come between the one you are trying to insert and the currently stored one.