Release Notes

Version 0.2.0

The September release is our largest update yet. We have added a lot of quality of life improvements and net new features to the RS360 product line. We plan on releasing even more features in our mid-October release to further improve the usability of the product as well as add additional features that will further help form the core of the product. A substantial amount of the features in this release were based directly off feedback from the first beta testing with external users and internal stakeholders. Please feel free to walk through our newest release. We welcome and encourage all constructive feedback on the product.

What’s New

  • Feature Table - After encouraging feedback from beta testing of the ID Stitcher, we are feeling more confident about sharing our C360 feature table functionality with beta customers. During testing of this release, we benchmarked ourselves against the feature set that our E-Commerce ML models expect. Many features were implemented successfully. Some needed functionality which could not be pushed through QA gates in this September release. Nevertheless, the feature table YAML is now ready for internal customers to explore.

  • Web App - We are now ready to share the scheduling functionality within the web app. This will allow the user to schedule, and automatically run WHT models from the Rudder backplane. Any artifacts and log files created during the execution of WHT projects are also available for the user to explore. This critical functionality will enable users to debug their cloud WHT runs.

  • Validate - A new command, wht validate allows users to run various tests on the project related configurations and validate the privileges associated with the role used for running the project. For example, the subcommand wht validate access does an exhaustive test of required privileges before accessing the warehouse.

  • Version - This is another new command that provides information on the current version of the app.

  • Logger - When you execute the compile and run commands, all errors and success messages that were previously only displayed on screen, are now also logged in a file inside the project output folder.

  • Discover - You can now export the output of the discover command in a CSV file. The ability to discover across all schemas in one’s warehouse is also added.

Improvements

  • We have made many changes to the way ID Stitcher config is written. We are forming a more complete opinion on the semantic model representation for customer’s data. Entities, IDs, and ID types are now defined in the WHT project file. The model file syntax is also more organized and easier to write. To see examples of the new syntax check out the section on Identity Stitching or sample files by executing command wht init wht-project. The sample project file also contains include and exclude filters, to illustrate their usage.

  • In wht command invocation, whenever a file is written, its location is now shown on the console and in log files.

  • Many enhancements on how errors are handled inside the application.

  • Massive improvements under the hood.

Bug Fixes

  • Fixed the issue in ID stitching where it was not picking up singleton components (i.e. the ones with only 1 edge), due to which they were getting skipped in the final output table.

  • In the init command, not entering any value for target wasn’t setting it to default value as “dev”.

  • Pressing Ctrl+C wasn’t exiting the application.

  • The command init profile now appends to an existing profile, instead of overwriting it.

  • Fixed the issue in discover command where the material table name was being displayed instead of the model name.

Known Issues:

  • Warning: While the run command is being executed, canceling it by pressing Ctrl+C doesn’t work as expected. Though it will stop the program’s execution on the CLI, the query will keep running on the data warehouse. This is a documented Snowflake behavior.

  • In a model, an input can’t use columns named “MAIN_ID”, “OTHER_ID”, “OTHER_ID_TYPE”, or “VALID_AT” in its ID SQL.

  • The web app is not showing a description and last run on the landing page.

  • In the web app, date filters to find wht runs aren’t working.

  • In the web app, when the WHT project is running, the screen shows an incorrect start timestamp.

  • Artifacts list changes when a project is running versus when it completes execution. Since all runs on the same Kubernetes pod share the same project folder, we are creating artifacts of different runs under the same parent folder. So, the same folder is currently shown for different runs of the project. In the next release, we will configure different folders for different runs to isolate them.

  • In case of feature table models, the compile command doesn’t always show error if the input SQL is buggy. Thise error may still be found when the model is run.

  • When creating a profile via init command, pressing the Ctrl+C command doesn’t exit the application.

  • Creating a WHT Project doesn’t currently include a sample independent ID stitcher. Instead, it is a child model to the generated feature table model.

  • We are working toward better readability of the logger file. We welcome any feedback here.

  • The command wht discover features needs to show a few more fields.

  • Every time wht run is executed for a feature table, it adds a new row to the output of wht discover features. Only one row should appear for each feature.

  • Export to CSV for the discover command should work for subcommands and also generate files in an output folder.

  • Some no-op parameters are shown upon passing the help flag(-h) to validate access command.

  • In some cases, error isn’t shown if an unknown flag is used.

  • Scheduling UI isn’t sometimes fully responsive about when the run will take place.

Note

The documentation for September release does not completely match with the current release. We are currently working on updating the documentation and will have new versions out soon. Please contact the Data Apps team if you are confused by some deviation.

Version 0.1.0

We are now in beta! Please do try out WHT and share your feedback with us, so that we can make it better.

What’s New

  • ID Stitcher - ID Stitching solves the problem of tying different identities together, for the same user across different sessions/devices. With v0.1.0, we launch WHT ID Stitching. It provides an easy to use and powerful interface to specify Id Stitching inputs.

  • Command Line Interface - Our CLI tool works on Linux, Windows and Mac machines. Using it you can setup a profile having connection to your Database, make a WHT project, create SQL from models, run ID stitcher models directly on the Warehouse, and discover all the created models/entities/sources on DW. [I have skipped Features as we will launch it in next version and will then give emphasis]

Improvements

  • We have enhanced the speed of Discover and Compile commands, from minutes to a few seconds.

  • The description of a few commands in Help has been improved.

Bug Fixes

  • The command for discovering entities wasn’t working, which has now been resolved.

  • Fixed the bug on init profile command where siteconfig wasn’t getting created on first-time installations.

  • A few bugs resolved related to output of discover command.

Known Issues:

  • Warning: While the run command is being executed, please do not cancel it by pressing Ctrl+C. Though it will stop the program’s execution on CLI, the query will keep running on the data warehouse. This is a documented Snowflake behaviour.

  • Null ID’s in ID stitcher. If first listed Id is null, the entire row may be ignored. That means, results are silently incorrect.

  • If first listed ID is null, the entire row may be ignored. The first listed ID is assumed to be the key ID. If it is ever null the results may be incorrect.