iModel Transformation and Data Exchange

The @bentley/imodeljs-backend package provides some classes that implement Extract, Transform, and Load (ETL) functionality:

The above classes contain the lower-level functionality required to implement transformation and data exchange services. These classes should be considered a framework and not confused with the actual packaged and deployed services that use the framework.

IModelExporter

The IModelExporter and IModelExportHandler base classes are used when the source data in an ETL workflow is contained within an iModel.

While it is possible to export data from an iModel using the standard IModelDb API, the IModelExporter and IModelExportHandler base classes offer the following capabilities:

Below is an example of using IModelExporter and IModelExportHandler to export all Code values from an iModel:

import { CodeSpec } from "@bentley/imodeljs-common";
import { Element, IModelDb, IModelExporter, IModelExportHandler, IModelJsFs as fs, SnapshotDb } from "@bentley/imodeljs-backend";

/** CodeExporter creates a CSV output file containing all Codes from the specified iModel. */
class CodeExporter extends IModelExportHandler {
  public outputFileName: string;

  /** Initiate the export of codes. */
  public static exportCodes(iModelDb: IModelDb, outputFileName: string): void {
    const exporter = new IModelExporter(iModelDb);
    const exportHandler = new CodeExporter(outputFileName);
    exporter.registerHandler(exportHandler);
    exporter.exportAll();
  }

  /** Construct a new CodeExporter */
  private constructor(outputFileName: string) {
    super();
    this.outputFileName = outputFileName;
  }

  /** Override of IModelExportHandler.onExportElement that outputs a line of a CSV file when the Element has a Code. */
  protected onExportElement(element: Element, isUpdate: boolean | undefined): void {
    const codeValue: string = element.code.getValue();
    if ("" !== codeValue) { // only output when Element has a Code
      const codeSpec: CodeSpec = element.iModel.codeSpecs.getById(element.code.spec);
      fs.appendFileSync(this.outputFileName, `${element.id}, ${codeSpec.name}, ${codeValue}\n`);
    }
    super.onExportElement(element, isUpdate);
  }
}

IModelImporter

The IModelImporter base class is used when the target in an ETL workflow is an iModel.

While it is possible to import data into an iModel using the standard IModelDb API, the IModelImporter class offers the following capabilities:

  • Callbacks whenever IModelImporter is used to insert, update, or delete entities. Simply override one of the protected onInsert*, onUpdate*, or onDelete* methods.
  • Optionally auto-extend IModel.projectExtents during import
  • Integration with IModelTransformer

IModelCloneContext

The IModelCloneContext class provides the core cloning capability required for iModel transformation. It also maintains the sourceId --> targetId mapping which is required to successfully clone Entity instances from the source iModel into the target iModel. iModel entities are highly related to each other. Therefore, cloning an entity means copying a graph of objects and remapping their source references (Ids) to other target entities.

IModelTransformer

The IModelTransformer base class is used when the source and target in an ETL workflow are both/different iModels and some sort of data transformation is needed in the middle. An instance of IModelTransformer holds instances of IModelExporter, IModelImporter, and IModelCloneContext. This means that customization is possible at the export stage, the transformation stage, and the import stage of the overall ETL process.

Potential transformations include:

  • Cloning - copying an entity from the source and remapping its internal Ids for the target
  • Filtering - excluding data from the target that is contained in the source
  • Augmenting - generating data during transformation for the target that is not part of the source
  • Schema Mapping - mapping classes and properties to a new schema during transformation
  • Change Squashing - each iModel has its own change ledger, so multiple changesets from the source could be squashed into a single changeset to the target

Logging

With batch processes like iModel transformation and data exchange, logging is often the only way to figure out what is actually happening. The following logger categories are provided for use with the Logger:

Last Updated: 17 September, 2020