As part of their manufacturing and testing process, this multi-locational client collected massive amounts of data at their different testing facilities around the country. Before DMC’s involvement, this data was automatically sent and saved to networked drives, which allowed the test data to be accessed by engineers at their other locations. However, the data files were large, all in different file formats, and not easily searchable on the drives. This required the data engineers to spend extensive amounts of time to manually sort through the data, identify relevant portions for analysis, and reformat it to make sure it can be exported in an easy-to-use file format.
To make this process more efficient while maintaining high levels of data security, DMC leveraged SystemLink, a software developed by National Instruments for data processing and management. As an NI Alliance Partner, DMC’s Test and Measurement team is very experienced in designing and integrating NI software for data management. SystemLink runs on a centralized server and can manage data from many different network locations while providing users with an internet browser user interface to search and access the managed data. This software also has a wide variety of other powerful data management tools, and this case study will focus on those tools’ abilities to automatically prepare, standardize, and index data.
Data Preparation and Standardization Using DIAdem
DMC began by automating the data preparation process with the help of DIAdem, another software created by National Instruments for managing large data. Similar to SystemLink, DIAdem is a robust software with many tools at your disposal. DIAdem provides advanced tools to edit automated scripts and queries that can be uploaded to the SystemLink server. For example, scripts written in Python or Visual Basic in DIAdem can be run to automate repetitive tasks once it’s uploaded to the server. For our client, we wrote a data plugin to solve the issue of inconsistent file formats by automatically transforming the files into a universal TDM format once they reach the SystemLink server.
The TDM file format, also developed by NI, standardizes measurement data into an accessible format. By transforming the data into a TDM format, the plugin eliminates time spent by the client preparing the data manually. The TDM file format also allows metadata to be associated to various data groups allowing users to easily search and query for files based on that metadata. For example, if a channel is created on January 1st, you can tag that timestamp to the data—along with the author, location, and other pertinent information. Those tags then become searchable through SystemLink’s data navigation feature. This allows SystemLink to essentially become a search engine for your data files.
The standardized TDM file format also allows for more efficient and automated post-processing. Just like the preparation and indexing capabilities of SystemLink, once data is sent to the server, it can be automatically analyzed and processed through scripting. For this particular client, we automated the calculation of channel metrics such as average and standard deviation for all data files. We also provided them with scripts that can be manually triggered to calculate ratios between channels. We customized the software for our client according to their current data processing needs, as well as providing them with a future-facing roadmap so their SystemLink server can change and grow with them.
Accessibility Without Compromising Security
After all the raw data was prepared, ingested, and indexed on the centralized SystemLink server, users can access the test data by logging into SystemLink’s web browser UI. The SystemLink security module allows the admin of the server to link LDAP or windows users and groups to SystemLink’s user permissions settings to authenticate and restrict web browser access as needed. This web browser UI makes remote access to the test data easy and secure by leveraging existing IT security configurations. On top of securing access to the web UI, SystemLink’s individual tools and features can also be assigned different permission levels via a concept called “Workspaces.” This makes it possible for the client to fully customize who can access SystemLink’s different tools and features. SystemLink’s web browser UI is an extremely convenient and powerful interface that provides users with remote access to SystemLink’s data and tools while maintaining security by linking to existing IT infrastructure for authentication.
SystemLink and DIAdem work hand-in-hand to make managing large sets of data easy. Diadem allows test engineers to develop data management scripts that can be uploaded to SystemLink for automated execution. By integrating DMC’s customized software into their process, our client was able to greatly save on labor costs and time spent performing repetitive and time-consuming tasks such as data mining and formatting, while still maintaining quality and security with their data workflow.
Learn more about DMC’s Test and Measurement Automation expertise and contact us to get started on your next project.