{"id":64004,"date":"2022-04-30T09:01:58","date_gmt":"2022-04-30T09:01:58","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=64004"},"modified":"2022-04-30T09:01:58","modified_gmt":"2022-04-30T09:01:58","slug":"parsing-data-with-strtok-in-c","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=64004","title":{"rendered":"Parsing data with strtok in C"},"content":{"rendered":"

Parsing data with strtok in C<\/span>
\nJim Hall<\/a><\/span>
\nSat, 04\/30\/2022 – 03:00<\/span><\/p>\n

\n
Register<\/a> or Login<\/a> to like<\/div>\n
Register<\/a> or Login<\/a> to like<\/div>\n<\/div>\n
\n

Some programs can just process an entire file at once, and other programs need to examine the file line-by-line. In the latter case, you likely need to parse data in each line. Fortunately, the C programming language has a standard C library function to do just that.<\/p>\n

The strtok<\/strong> function breaks up a line of data according to “delimiters” that divide each field. It provides a streamlined way to parse data from an input string.<\/p>\n

Reading the first token<\/h2>\n

Suppose your program needs to read a data file, where each line is separated into different fields with a semicolon. For example, one line from the data file might look like this:<\/p>\n

\n102<\/span>*<\/span>103<\/span>;<\/span>K1.2;<\/span>K0.5<\/code><\/span><\/pre>\n
In this example, store that in a string variable. You might have read this string into memory using any number of methods. Here’s the line of code:<\/p>\n
\nchar<\/span> string[<\/span>]<\/span> =<\/span> \"102*103;K1.2;K0.5\"<\/span>;<\/span><\/code><\/span><\/pre>\n
Once you have the line in a string, you can use strtok<\/strong> to pull out “tokens.” Each token is part of the string, up to the next delimiter. The basic call to strtok<\/strong> looks like this:<\/p>\n
\n
#include <\/string.h><\/span>
char<\/span> *<\/span>strtok<\/span><\/a>(<\/span>char<\/span> *<\/span>string,<\/span> const<\/span> char<\/span> *<\/span>delim)<\/span>;<\/span><\/div><\/div><\/pre>\n
The first call to strtok<\/strong> reads the string, adds a null (\u0000<\/code>) character at the first delimiter, then returns a pointer to the first token. If the string is already empty, strtok<\/strong> returns NULL.<\/p>\n
\n
#include <\/stdio.h><\/span>
#include <\/string.h><\/span>

int<\/span>
\nmain(<\/span>)<\/span>
{<\/span>
\n\u00a0 char<\/span> string[<\/span>]<\/span> =<\/span> \"102*103;K1.2;K0.5\"<\/span>;<\/span>
\n\u00a0 char<\/span> *<\/span>token;<\/span>

\n\u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>string,<\/span> \";\"<\/span>)<\/span>;<\/span>

\n\u00a0 if<\/span> (<\/span>token ==<\/span> NULL)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>\"empty string!\"<\/span>)<\/span>;<\/span>
\n\u00a0 \u00a0 return<\/span> 1<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 puts<\/span><\/a>(<\/span>token)<\/span>;<\/span>

\n\u00a0 return<\/span> 0<\/span>;<\/span>
}<\/span><\/div><\/div><\/pre>\n
This sample program pulls off the first token in the string, prints it, and exits. If you compile this program and run it, you should see this output:<\/p>\n
\n102<\/span>*<\/span>103<\/span><\/code><\/span><\/pre>\n
102*103 is the first part of the input string, up to the first semicolon. That’s the first token in the string.<\/p>\n
Note that calling strtok<\/strong> modifies the string you are examining. If you want the original string preserved, make a copy before using strtok<\/strong>.<\/p>\n<\/p>\n
\n
Programming and development<\/div>\n
\n
Red Hat Developers Blog<\/a><\/div>\n
Programming cheat sheets<\/a><\/div>\n
Try for free: Red Hat Learning Subscription<\/a><\/div>\n
eBook: An introduction to programming with Bash<\/a><\/div>\n
Bash shell scripting cheat sheet<\/a><\/div>\n
eBook: Modernizing Enterprise Java<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n
Reading the rest of the string as tokens<\/h2>\n
Separating the rest of the string into tokens requires calling strtok<\/strong> multiple times until all tokens are read. After parsing the first token with strtok<\/strong>, any further calls to strtok<\/strong> must use NULL in place of the string variable. The NULL allows strtok<\/strong> to use an internal pointer to the next position in the string.<\/p>\n
Modify the sample program to read the rest of the string as tokens. Use a while loop to call strtok<\/strong> multiple times until you get NULL.<\/p>\n
\n
#include <\/stdio.h><\/span>
#include <\/string.h><\/span>

int<\/span>
\nmain(<\/span>)<\/span>
{<\/span>
\n\u00a0 char<\/span> string[<\/span>]<\/span> =<\/span> \"102*103;K1.2;K0.5\"<\/span>;<\/span>
\n\u00a0 char<\/span> *<\/span>token;<\/span>

\n\u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>string,<\/span> \";\"<\/span>)<\/span>;<\/span>

\n\u00a0 if<\/span> (<\/span>token ==<\/span> NULL)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>\"empty string!\"<\/span>)<\/span>;<\/span>
\n\u00a0 \u00a0 return<\/span> 1<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 while<\/span> (<\/span>token)<\/span> {<\/span>
\n\u00a0 \u00a0 \/* print the token *\/<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>token)<\/span>;<\/span>

\n\u00a0 \u00a0 \/* parse the same string again *\/<\/span>
\n\u00a0 \u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>NULL,<\/span> \";\"<\/span>)<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 return<\/span> 0<\/span>;<\/span>
}<\/span><\/div><\/div><\/pre>\n
By adding the while loop, you can parse the rest of the string, one token at a time. If you compile and run this sample program, you should see each token printed on a separate line, like this:<\/p>\n
\n
102<\/span>*<\/span>103<\/span>
\nK1.2
\nK0.5<\/div><\/div><\/pre>\n
Multiple delimiters in the input string<\/h2>\n
Using strtok<\/strong> provides a quick and easy way to break up a string into just the parts you’re looking for. You can use strtok<\/strong> to parse all kinds of data, from plain text files to complex data. However, be careful that multiple delimiters next to each other are the same as one delimiter.<\/p>\n
For example, if you were reading CSV data (comma-separated values, such as data from a spreadsheet), you might expect a list of four numbers to look like this:<\/p>\n
\n1<\/span>,<\/span>2<\/span>,<\/span>3<\/span>,<\/span>4<\/span><\/code><\/span><\/pre>\n
But if the third “column” in the data was empty, the CSV might instead look like this:<\/p>\n
\n1<\/span>,<\/span>2<\/span>,,<\/span>4<\/span><\/code><\/span><\/pre>\n
This is where you need to be careful with strtok<\/strong>. With strtok<\/strong>, multiple delimiters next to each other are the same as a single delimiter. You can see this by modifying the sample program to call strtok<\/strong> with a comma delimiter:<\/p>\n
\n
#include <\/stdio.h><\/span>
#include <\/string.h><\/span>

int<\/span>
\nmain(<\/span>)<\/span>
{<\/span>
\n\u00a0 char<\/span> string[<\/span>]<\/span> =<\/span> \"1,2,,4\"<\/span>;<\/span>
\n\u00a0 char<\/span> *<\/span>token;<\/span>

\n\u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>string,<\/span> \",\"<\/span>)<\/span>;<\/span>

\n\u00a0 if<\/span> (<\/span>token ==<\/span> NULL)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>\"empty string!\"<\/span>)<\/span>;<\/span>
\n\u00a0 \u00a0 return<\/span> 1<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 while<\/span> (<\/span>token)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>token)<\/span>;<\/span>
\n\u00a0 \u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>NULL,<\/span> \",\"<\/span>)<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 return<\/span> 0<\/span>;<\/span>
}<\/span><\/div><\/div><\/pre>\n
If you compile and run this new program, you’ll see strtok<\/strong> interprets the ,,<\/code> as a single comma and parses the data as three numbers:<\/p>\n
\n
1<\/span>
2<\/span>
4<\/span><\/div><\/div><\/pre>\n
Knowing this limitation in strtok<\/strong> can save you hours of debugging.<\/p>\n
Using multiple delimiters in strtok<\/h2>\n
You might wonder why the strtok<\/strong> function uses a string for the delimiter instead of a single character. That’s because strtok<\/strong> can look for different delimiters in the string. For example, a string of text might have spaces and tabs between each word. In this case, you would use each of those “whitespace” characters as delimiters:<\/p>\n
\n
#include <\/stdio.h><\/span>
#include <\/string.h><\/span>

int<\/span>
\nmain(<\/span>)<\/span>
{<\/span>
\n\u00a0 char<\/span> string[<\/span>]<\/span> =<\/span> \" \u00a0hello t<\/span> world\"<\/span>;<\/span>
\n\u00a0 char<\/span> *<\/span>token;<\/span>

\n\u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>string,<\/span> \" t<\/span>\"<\/span>)<\/span>;<\/span>

\n\u00a0 if<\/span> (<\/span>token ==<\/span> NULL)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>\"empty string\"<\/span>)<\/span>;<\/span>
\n\u00a0 \u00a0 return<\/span> 1<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 while<\/span> (<\/span>token)<\/span> {<\/span>
\n\u00a0 \u00a0 puts<\/span><\/a>(<\/span>token)<\/span>;<\/span>
\n\u00a0 \u00a0 token =<\/span> strtok<\/span><\/a>(<\/span>NULL,<\/span> \" t<\/span>\"<\/span>)<\/span>;<\/span>
\n\u00a0 }<\/span>

\n\u00a0 return<\/span> 0<\/span>;<\/span>
}<\/span><\/div><\/div><\/pre>\n
Each call to strtok<\/strong> uses both a space and tab character as the delimiter string, allowing strtok<\/strong> to parse the line correctly into two tokens.<\/p>\n
Wrap up<\/h2>\n
The strtok<\/strong> function is a handy way to read and interpret data from strings. Use it in your next project to simplify how you read data into your program.<\/p>\n<\/div>\n
\n
The strtok function is a handy way to read and interpret data from strings. Use it in your next project to simplify how you read data into your program.<\/p>\n<\/div>\n
\n
\n
  \"Woman<\/div>\n
Image by: <\/span><\/p>\n
kris kr\u00fcg<\/p>\n<\/div>\n<\/article>\n<\/div>\n
\n
Programming<\/a><\/div>\n<\/p><\/div>\n
What to read next<\/div>\n

\n        \"Creative<\/a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.<\/div>\n
\n
\n
Register<\/a> or Login<\/a> to post a comment.<\/div>\n<\/p><\/div>\n<\/section>\n
Powered by WPeMatico<\/a><\/small><\/p>\n","protected":false},"excerpt":{"rendered":"
Parsing data with strtok in C Jim Hall Sat, 04\/30\/2022 – 03:00 Register or Login to like Register or Login to like Some programs can just process an entire file at once, and other programs need to examine the file line-by-line. In the latter case, you likely need to parse data in each line. Fortunately, […]<\/p>\n","protected":false},"author":1,"featured_media":64005,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[307],"tags":[],"class_list":["post-64004","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-open-source"],"_links":{"self":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/64004","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=64004"}],"version-history":[{"count":0,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/64004\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/media\/64005"}],"wp:attachment":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=64004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=64004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=64004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}