Blog

LabVIEW Data Storage: Overview of TDMS

LabVIEW Data Storage: Overview of TDMS

This is part one of my blog series detailing the use and optimization of the TDMS format. Here is a table of contents for all blogs in this series:

LabVIEW Data Storage: Overview of TDMS

LabVIEW Data Storage: TDMS Performance Tweaking

LabVIEW Data Storage: TDMS Usage Case Study

Here at DMC, almost all of our LabVIEW applications acquire data at one point or another. This data often needs to be saved to disk for later review, display, and/or exported to a report. Years ago, logging to disk would mean working with our client to decide what type of file format fit their needs the best.

If they were concerned about disk space and needed high speed streaming rates we would use a raw binary file or other binary format like NI's datalog. If they wanted the data available for review and calculations in Excel we would create a CSV file with the trade-off being speed and large log files. If they were simply looking to save settings or setup data we would use an INI file format. In the last couple of years though, NI has released a file format called TDM which was later expanded by a format called TDM Streaming or TDMS.

TDMS is described fairly in depth on National Instrument's site (NI TDM Data Model). Essentially, it's a file that includes a combination of metadata strings for things like names/properties/hierarchy and binary data for higher throughput, more byte consuming raw data. This model gives it the flexibility of an INI or XML file but with the high speed access and efficiency of a pure binary file.

TDMS Hierarchy

Because of the great utility of TDMS, we find ourselves now using it on nearly every LabVIEW project, even if it's just to save some data during the debug phase.

To get started with TDMS, search the LabVIEW examples and start with the simplified "Write TDMS File.vi" and "Read TDMS File.vi". You can easily access the TDMS functions from the LabVIEW Palette. Just look in the File I/O submenu.

TDMS in LabVIEW Palette

My next blog post will focus on TDMS performance and what's happening behind the scenes. Before going much further, I recommend familiarizing youself with with the concept of Groups, Channels, and Properties and how they can be used within the TDMS Hierarchy. Either look through the NI site on this subject (NI TDM Data Model) or play around with the example VIs until you have a good feel for what's going on.

The next blog in this series isLabVIEW Data Storage: TDMS Performance Tweaking

Learn more about DMC's data analysis, data mining and reporting expertise.

Comments

Jody Koplo
# Jody Koplo
Tushar,

The max streaming speed of TDMS is pretty variable. On RAID arrays, using some clever optimized writing schemes, NI has benchmarked TDMS streaming at rates of 1.6 Gigabytes/second. Obviously, your streaming rate is very dependent on the speed of your storage device.

I would say that in general, TDMS is going to stream data faster than anything but a custom binary format. It's definitely going to beat a text/xml file.

As for reading data by random access, that's one of the strongest strengths of TDMS. Check out my next article that talks a lot about optimizing and using TDMS files: http://www.dmcinfo.com/blog.aspx/articleType/ArticleView/articleId/205/LabVIEW-Data-Storage-TDMS-Performance-Tweaking.aspx
Tushar Jambhekar
1. whats the approx. max streaming speed achievable if we use TDMS format.

2. can we use this file format if we want to randomly access data afterward
e.g. if i am logging 4 channels at 1ksps and if i want to read 3rd ch data for time 10th second to 12th second. do i need to read entire file first?

Post a comment

Name (required)

Email (required)

CAPTCHA image
Enter the code shown above: