[hemmerling] Data Processing 6/8 - Data Warehouse Modeling

Related pages:

• Data Processing 1/8 - Databases, Webserver, Webservices, SQL, XML.
• Data Processing 2/8 - Descriptions of Data, Data Transfer and Data Protocols.
• Data Processing 3/8 - NoSQL Databases: Not Just SQL.
• Data Processing 4/8 - Database Managment Tools and Database Application Development Tools.
• Data Processing 5/8 - Business Intelligence.
• Data Processing 7/8 - Machine Learning.
• Data Processing 8/8 - Artificial Intelligence ( AI ).

1. Anchor
   • EN.Wikipedia "Anchor modeling".
2. Data Vault.
3. Focal Point.
4. Hyper Agility.

• Genesee Academy, LLC.
  • “Our mission is to provide world class training and certification for data warehousing, data vault, ensemble modeling, data integration and business intelligence professionals”.
  • Hans Hultgren is one of the lecturers .
• YouTube "DataVaultAcademy".

• LinkedIn "Hans Hultgren".
• Blog The Hans Blog.
  • The Hans Blog "Hyper Agility".
  • The Hans Blog "Focal Point Modeling".
• YouTube - Search for "Hans Hultgren".
• Twitter "Hans Hultgren, @gohansgo".
• Book Hans Hultgren "Modeling the Agile Data Warehouse with Data Vault", 2012.
• Blog Hans Hultgren's DataVaultBook Blog "Modeling the Agile Data Warehouse with Data Vault".
• Experts told me, that Hans has different opinions and vocabulary than Dan Linstedt & Michael Olschimke .

• EN.Wikipedia "Bill Inmon", DE.Wikipedia "Bill Inmon".

• Dan Linstedt.
  • LinkedIn "Dan Linstedt".
  • Dan Linstedt "#datavault model and understanding, back to basics".
  • LearnDataVault.
  • YouTube "learndatavault".
  • LinkedIn "Learn Data Vault".
• Twitter "Daniel Linstedt, @dlinstedt".

• Michael Olschimke
  • XING "Michael Olschimke", LinkedIn "Michael Olschimke".
  • Commpanies:
    • Dörffler & Partner GmbH.
    • Scalefree - Olschimke Beratungsgesellschaft mbH - “The Original Data Vault 2.0 Training”.

• Visual Data Vault.
• “In collaboration with Dan Linstedt, we have developed a visual modeling language that facilitates to represent a Data Vault model graphically: Visual Data Vault. It’s a logical modeling language with which you can model hubs, links, satellites, and other Data Vault entities. All icons are standardized and easy to learn. A simple model can be depicted with a few symbols”.
• “To make this notation easily available to members of any Data Vault project team, we have developed a 2010+ stencil for Microsoft Visio. This can be requested via the form on this page. It can be used in projects free of charge”.

• Book Dan Linstedt, Michael Olschimke "Building a Scalable Data Warehouse with Data Vault 2.0", 2015.
• Book W. H. Inmon, Daniel Linstedt "Data Architecture: A Primer for the Data Scientist: Big Data, Data Warehouse and Data Vault", 2014.
• Book Kent Graziano, Dan Linstedt "Super Charge Your Data Warehouse: Invaluable Data Modeling Rules to Implement Your Data Vault", 2011.

• Blog Hans Michiels - Blog and more.
  • Hans Michiels "The Data Warehouse Automation Tools Survey Results (Data Warehouse Series)" - List of Data Warehouse Automation tools #.
• Datawarehouse automation Guide.
• Twitter "Hans Michiels, @dwaguide".

• Book Len Silverston: "The Data Model Resource Book: A Library of Universal Data Models for All Enterprises, Volume 1".
  • CD-ROM Len Silverston: "The Data Model Resource Book: A Library of Universal Data Models for All Enterprises, Volume 2".
• Book Len Silverston: "The Data Model Resource Book: A Library of Universal Data Models by Industry Types, Volume 2".
  • CD-ROM Len Silverston: "The Data Model Resource Book: A Library of Universal Data Models by Industry Types, Volume 2".
• Book Len Silverston, Paul Agnew: "The Data Model Resource Book: A Library of Universal Data Models for All Enterprises, Volume 3".
• I was told, that the official “Microsoft SQL server CD” was shipped with some licensed data models by Len Silverston.

• Cathrine Wilhelmsen - “Biml, SQL, Speaking and Volunteering”.
• Cathrine Wilhelmsen "Biml Resources and References".
• Twitter "Cathrine Wilhelmsen, @cathrinew" - “DWH/BI/SQL geek, Data Platform MVP, BimlHero, author, speaker, blogger and chronic volunteer. Loves sci-fi, chocolate, cat gifs and smilies :)”.

• CloverETL by Javlin a.s. - “Rapid Data Integration Software and Open Source ETL Tool”.
  • CloverETL "Open Source & Community Edition".
  • CloverETL "CloverETL Community Edition".
  • Blog ClovertETL Blog "CloverETL Blog on Data Integration".
• GitHub "CloverETL".
• SourceForge "CloverETL", SourceForge "CloverETL" - “Rapid data integration framework”.

• Varigence.
  • The commercial software Varigence "BimlStudio. The only integrated development environment for Biml" ( formerly: Mist ).
  • The free software Varigence "BimlExpress. Visual Studio Add-in".
  • Varigence "Biml" - “Business Intelligence Markup Language. Human readable and writable XML language for automating and creating business intelligence solutions”.

• BimlScript - “Learn Biml Now! If you’re new to Biml, start by learning how it can fit into your data development workflow”.
• BimlAcademy.
  • BimlAcademy "Lesson 8 – BimlFlex for Data Vault Data Warehouses – Scott Currie".
• Varigence Documentation "Biml Snippets".
  • Varigence Documentation "Varigence Library" - “Biml Language”, “Biml Api”]].
• YouTube Channel "BimlScript".

• The OpenSource framework for SQL server CodePlex "BIDS Helper" - “A Visual Studio add-in that enhances development functionality in Business Intelligence Development Studio (BIDS) and SQL Server Data Tools (SSDTBI)”.
• The legacy OpenSource framework for SQL server CodePlex "Vulcan" - “Vulcan is a Business Intelligence Framework built on top of Sql Server Integration Services and Sql Server Analysis Services for quickly creating BI solutions”.
• GitHub.io "BI Developer Extensions", GitHub "BI Developer Extensions" - “A Visual Studio add-in that enhances development functionality in Business Intelligence Development Studio (BIDS) and SQL Server Data Tools (SSDTBI)”.
• VisualStudio Marketplace "BI Developer Extensions for Visual Studio 2017".
  • “BI Developer Extensions (formerly BIDS Helper) is an extension for BIDS / SSDT that includes numerous enhancements for SQL Server BI projects”.
  • “A Visual Studio add-in that enhances development functionality in SQL Server Data Tools (SSDT)”.
• EN.Wikipedia "BIDS Helper".

• Varigence "BimlFlex".
  • “A flexible, extensible, and completely customizable data warehouse framework”.
  • “BimlFlex is a collection of templates, metadata definitions, and tooling that enables you to build

an end-to-end data solution without ever writing a single line of code”.

• YouTube "Varigence BimlFlex data warehouse automation framework overview".

• Experts told me “Methodology: Dimensional(?)”.
• Experts told me:
  • In case that a table doesn´t exist, BIMLStudio has the feature “Offline Scheme”. You may query the database as if the table would exist.
  • Even With BIMLExpress, you may generate this “Offline Scheme” by 2 ways:
    • Manually, from “Meta Data”.
    • With Visual Studio, the developer gets access to the database, opens it and reads the scheme, without copying the datebase.
• Samudra Bandaranayake "BIML Feature Comparison".
  • BIMLExpress – Free Plugin for Visual Studio
  • BimlStudio – (The software formerly known as Mist)
• AndyLeonard.Blog() "So You Want to Learn More About Biml?", SQL Blog, Andy Leonard "So You Want to Learn More About Biml?", 2016.
• Hans Michiels – Blog and more "11 Things you should know about Biml – an overview (Datawarehouse Series)".
• SQL Server Central "Stairways to BIML".
• Blog Roelant Vos - Blog.
• EN.Wikipedia "Business Intelligence Markup Language".
• The discontinued EN.Wikipedia "Business Intelligence Development Studio".
• EN.Wikipedia "Varigence".

• CodePlex "BI Builder for SQL Server" - “A pure T-SQL application for MSSQL data warehouse developers: It generates staging and dimension tables as well as ETL stored procedures”.
• GitHub "CuriousDevelopments/bibuilder".
• Experts told me “Methodology: Dimensional”.

• Optimal Data Engine ( ODE ).
  • Optimal Data Engine ( ODE ) "3CpO" - “3CpO is the name we give to the ODE framework. Its an acronym for three C’s plus an O”:
    1. Configuration
    2. Code Generation
    3. Code Execution
    4. Orchestration
  • “The core of ODE is built around the Data Vault approach”.
• GitHub "Optimal Business Intelligence" - “We Make Information Visible, you make informed decisions”.
• Experts told me “Methodology: Data Vault”.

• Qosqo - “Quipu Software”. “Data Warehouse Automation”.
• Quipu Software - Data Warehouse model generation ( datawarehousemanagement.org ) - “Quipu software”.
• Theobald Software "QUIPU Data Warehouse Management".
• YouTube "Quipu DWM".
• Twitter "QOSQO, @Quipu_DWM".
• Nippur Zuid - “Our sister company Nippur assists in executing business intelligence projects”.

• SourceForge "Quipu", SourceForge "Quipu", a Java6 application.
• “Quipu is an open source data warehouse generation system that creates and monitors data warehouses. With Quipu you can implement a data warehouse much quicker and easier”.
• “In 2015, the software company Qosqo who supported the development of Quipu decided to abandon the open source project as it was. The latest open source release (version 2.x) is what you'll find here”.

• Quipu - “Invoicing software for enterprises, freelances and bookkeepers”.
• WordPress "Quipu for WooCommerce".
• OpenSourceDevelopmentNetwork ( OSDN ) "quipu" - “Quipu is a new programming language of the lisp family with a focus on concurrency, lock-free data structures and expresiveness”.
• Quipu Project, MIT _docubase "_Quipu Project".
• EN.Wikipedia "ISO Development Environment" - “Quipu”.
• EN.Wikipedia "Quipu", DE.Wikipedia "Quipu".

• Experts told me “Methodology: Data Vault”.
• b-eye Blog, Ronald Damhof "Agile Data Warehousing - Ralph Hughes visits Data Vault customers in Amsterdam", 2011 - “BinckBank used Quipu for data warehouse automation and the data warehouse was partly driven by business key, and some by surrogate key”.
• BI Future Blog - Search for "Quipu" - Experts told me that this is an important blog about “Quipu”.
• BI-Platform - Search for "Quipu".
  • BI-Platform "Open Source dataware-housing van eigen bodem" ( PDF of “Database Magazine – Nummer 4 – juni 2010” ) - “Currently a community edition (CE) of QUIPU launched... This edition... has no restrictions on size or complexity of the data model, the number of data models and the data volume that the generated data house”.
• b.telligent Blog "Data Warehouse Automation (Part 1)" - “QOSQO quipu”.
• Roland Bouman's blog "Back to blogging....", 2010 - “I have been checking out Quipu, a promising Open Source data warehouse management solution. Quipu provides a repository-based extensible code-generator that allows you to generate and maintain a data warehouse based on the Data Vault model. One of the things I want to do in the short term is write templates that allows it to work for MySQL, and after that, I want to see if I can get Quipu to generate Kettle Jobs and transformations”.
• Data Warehouse Automation - Search for "Quipu".
  • Data Warehouse Automation "Loading your data vault", 2013.
  • Data Warehouse Automation "Data warehouse generation algorithm explained", 2013.
• Linux Questions "Open source and Data Vault modelling", 2010 - “Quipu”.
• Ondernemend Friesland nieuws "Binckbank selecteert Quipu", 2011-03-30 - “After an extensive selection process, BinckBank has opted for the open source software Quipu. In the coming months BinckBank will continue to develop Quipu together with QOSQO (the developer of Quipu located in Oirschot). Afterwards, this version will also be made publicly available in accordance with the open source philosophy... Since the launch on July 1, 2010 there are (as far as is known) in the Netherlands three multinationals and a government agency that use Quipu. Worldwide, more than 600 registered users have downloaded the software.”.
• Nippur Zuid "Artikelen" - Article “Quipu: open source datawarehouse management” ( access has to be requested by formular ).
• Qubiz "Data vaults - products that enable you to work easier" - Section “Quipu – data warehouse management”.
• Scribd, Jan Jansen "Quipu Data Warehouse Management".
• SlideShare "Data Vault automation conference - all presentations", 2011 - “Quipu” ( slided 15 - 43 ).
• Roelant Vos Blog "Do we still want to automate against ETL tools?", - “I do realise there are various ETL automation tools out there that generate SQL. Software such as Quipu (open source), WhereScape, BIReady etc. etc. But as far as I know they don’t do the virtualisation for the core Data Warehouse layer yet, and in some cases (to the extend that I have worked with the tools) have some limitations that I’m not comfortable with”.

• RapidACE Automation ( - 2013-03-26 ).
• RapidACE Automation News "RapidACE – is Going Open Source!", 2012.
• Blog Archive.org "RapidACE Services - Architectural Consolidation Enablement".
• Twitter "rapidace" - “Account suspended. This account has been suspended” .
• Dan Linstedt - Articles tagged with "rapidace".
• Dan Linstedt "RapidACE – Going Open Source", 2011.
  • “I’ve decided…The RapidACE java product I’ve written is going to be placed in to open-source... But both version 1, and version 2 tools are complex – they are enterprise focused applications,with tons of configuration options... Version 1 has a number of hard-bugs (major design flaws), and v2 has NO GUI – it’s all 100% command line driven with XML configuration files (today)”.
• SourceForge Profile "dlinstedt".
  • SourceForge "RapidACE Templates", SourceForge "RapidACE Templates".
    • “For templates to use with RapidACE: Rapid Architectural Consolidation Engine (it is a commercial product). The templates generate code based on RapidACE cross-references. The templates are based on FreeMarker and FMPP template engines”.
    • “This project has no files” .
  • SourceForge "RapidACEOS", SourceForge "RapidACEOS" .
    • Archive.org "SourceForge 'RapidParse'", 2011-01-31.
      • “RapidParse: DDL (data definition lang) to XML converter. Reads text files, parses into a RapidACE XML format. It is 100% java command line, built for performance. Generic format parser for Teradata, MySQL, Oracle, DB2 UDB, Sybase, and SQLServer DDL”.
      • Features:
        • Parses DDL to RapidACE XML (open file)
        • Parses Teradata, MySQL, SQLServer, Db2 UDB, Oracle, and Sybase DDL
        • Handles Create, Drop, and Alter Commands
        • Manages comments, title, keys, indexes, and storage commands
        • Parses Alter Commands
        • Work in progress: parsing stored procs and SQL views
    • Archive.org "SourceForge 'RapidACE', 2013-01-25.
      • SourceForge SVN "RapidACE".
    • SourceForge Site Support #3920 "Remove RapidACE os" - “Please remove the project, I have had zero interest in maintaining, or updating the code. No other developers have volunteered code changes, and there have been no updates to the code base for over 3 years. It is time to remove this project in its entirety. Thank-you kindly, Dan Linstedt” .

• Twitter - Search for "#rapidace".
• In 2018-08, I didn´t find any download offer of the OpenSource software “RapidICE” .

• Experts suggested to create a domain specific language as base for own ( OpenSource ) tools. See Textual and Graphical Modeling by Domain Specific Languages.

• Ajilius - Data Warehouse Automation.
• Blog Ajilius Blog.
• Ajilius - Data Warehouse Automation "Competitors" - List of Data Warehouse Automation tools #.
• Experts told me “Methodology: Dimensional”.

• AnalytiX DS.
• Twitter "Dan Linstedt, @RapidGenDS" - “Creating new software for http://AnalytixDS.com Mapping Manager, in the area of Automation for Data Warehousing and Business Intelligence”.
• Archive.org "RapidGenDS - AnalytixDS & Mapping Manager" - “Please see AnalytiX DS for all future inquiries”.
• EN.Wikipedia "AnalytiX DS".
• Experts told me “Methodology: Data Vault, 3NF, Dimensional”.

• BI Ready "Attunity Compose Data Warehouse Automation".
• Attunity "Attunity Compose" - “Agile ETL and Data Warehouse Automation”.
• Blog Attunity "Data Management Blog".
• Experts told me “Methodology: Data Vault”.

• BI Builders "Xpert BI" - “Xpert BI is a Microsoft SQL server based tool that is designed to automate and improve all aspects of data warehousing and Business Intelligence – especially in the areas of extraction- and transformation processes, also known as ELT”.
• Experts told me “Methodology: Dimensional”.

• Birst, an Infor company.
  • “Modern cloud BI and analytics to optimize your most complex business processes”.
  • “Birst’s Networked Analytics and BI platform is a new and disruptive approach to cloud business analytics”.
• Blog Birst Blog "BI News, Business Intelligence Information".
• Experts told me “Methodology: Dimensional”.

• Centennium "Centennium Automation Tool".
• Experts told me “Methodology: Data Vault”.
• Website is in Dutch language, only.

• Datavault Builder.
  • “Data Warehousing. Simply Visual”.
  • “Next Generation Software for Visual Data Integration”.
• Experts told me “Methodology: Data Vault”.

• Dimodelo Solutions Pty "Dimodelo Architect".
• Experts told me “Methodology: Dimensional”.

• Gamma Systems, Inc. "Data Warehouse Studio".
• YouTube "GammaSystemsInc".
• Experts told me “Methodology: Dimensional”.

• Halo Business Intelligence.
• Experts told me “Methodology: Dimensional”.

• Insource Ltd. "Enterprise Data Management Platform Data Academy".
• Experts told me “Methodology: Dimensional”.

• LeapFrogBI - “The LeapFrogBI Platform. Hyper-fast SQL Server ETL development”.
• Experts told me “Methodology: Dimensional”.

• Magnitude Software, Inc..
• YouTube "Magnitude Software".
• Experts told me “Methodology: Dimensional”.

• OceanBI "DDM Studio".
• Experts told me “Methodology: Data Vault and Dimensional”.

• TimeXtender – “Enabling instant access to data through Discovery Hub”.
• Experts told me “Methodology: Dimensional”.

• Trivadis "biGenius".
• Experts told me “Methodology: Unknown”.

• WhereScape.
  • WhereScape "WhereScape RED" - “Deliver data infrastructure more quickly with less hassle. Automate to develop, deploy, and operate data infrastructure and big data integration projects the fast, easy way”.
  • WhereScape "WhereScape Data Vault Express" - “Automate the entire Data Vault life cycle for faster time to production, lower cost, and greater success”.
• I joined the free 3-hours hand-on workshop “WhereScape RED Test Drive” at the conference TDWI Konferenz 2018 ( 2018-06-25 - 2018-06-27 ) in München, Germany.
• Experts told me “Methodology: Dimensional, Data Vault(?)”.
• EN.Wikipedia "WhereScape".

• Matt Casters on Data Integration. Coding, Kettle, Pentaho, Big Data and more "Data Modeling", 2011 - “However, where these days you have open source tools like Quipu and RapidACE for data vault modelling I was sad to see that not too much exists for dimensional modeling in combination with Kettle”.
• Estrenuo – All about data "The 'catch' with data warehouse automation tools", 2012 - “During my evaluation of several data warehouse automation tools such as BIReady, Quipu and RapidAce, I have come to some sort of conclusion that is crucial to the success of using these tools”.
• Dan Linstedt "Data Vault / ETL /ELT / Ontologies and Data Models", 2010.
  • “For a long time I had a feature in RapidACE called Model Consolidation”.
  • ”(opensource Quipu product, Model Engineering + ETL Generation)”

• The bachelor thesis as printed book Christian Garbsch, Dominik Kroner "Business Intelligence Automatisierung durch Data Vault: Umsetzungen mit Microsoft Sql Server Integration Services und Sas Base", 2014.

• There are 3 entities: Hub, Link, Satellite ( of 57 originally selected entities ).
• EN.Wikipedia "Data Vault Modeling", DE.Wikipedia "Data Vault".
• Reddit "Data Warehouse Automation Tools".

• EN.Wikipedia "Data warehouse automation".
• EN.Wikipedia "Common Warehouse Metamodel", DE.Wikipedia "Common Warehouse Metamodel" ( CWM ).