Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Word Frequency Statistics

In an pre-interview, I am faced with a question like this:

Given a string consists of words separated by a single white space, print out the words in descending order sorted by the number of times they appear in the string.

For example an input string of “a b b” would generate the following output:

b : 2
a : 1

Firstly, I'd say it is not so clear that whether the input string is made up of single-letter words or multiple-letter words. If the former is the case, it could be simple.

Here is my thought:

int c[26] = {0};
char *pIn = strIn;

while (*pIn != 0 && *pIn != ' ')
{
    ++c[*pIn];
    ++pIn;
}

/* how to sort the array c[26] and remember the original index? */

I can get the statistics of the frequecy of every single-letter word in the input string, and I can get it sorted (using QuickSort or whatever). But after the count array is sorted, how to get the single-letter word associated with the count so that I can print them out in pair later?

If the input string is made of of multiple-letter word, I plan to use a map<const char *, int> to track the frequency. But again, how to sort the map's key-value pair?

The question is in C or C++, and any suggestion is welcome.

Thanks!

like image 875
Qiang Xu Avatar asked Dec 30 '11 15:12

Qiang Xu


People also ask

How is word frequency calculated?

Count the number of times you use everyword word in a text.

What is word frequency analysis?

Frequency analysis can also be used as a means to establish the “signature” of a certain author, the cultural level of the writer, its use of slang or technical jargon, and other writing features. It is possible to extrapolate the number of words used in a certain text to the total vocabulary of a person.

What is word frequency in NLP?

After tokenising a text, the first figure we can calculate is the word frequency. By word frequency we indicate the number of times each token occurs in a text.

How does word frequency affect word recognition?

When word recognition is analyzed, frequency of occurrence is one of the strongest predictors of processing efficiency. High-frequency words are known to more people and are processed faster than low-frequency words (the word frequency effect; Monsell, Doyle, & Haggard, 1989).


2 Answers

I would use a std::map<std::string, int> to store the words and their counts. Then I would use something this to get the words:

while(std::cin >> word) {
    // increment map's count for that word
}

finally, you just need to figure out how to print them in order of frequency, I'll leave that as an exercise for you.

like image 159
Evan Teran Avatar answered Oct 17 '22 01:10

Evan Teran


You're definitely wrong in assuming that you need only 26 options, 'cause your employer will want to allow multiple-character words as well (and maybe even numbers?).

This means you're going to need an array with a variable length. I strongly recommend using a vector or, even better, a map.

To find the character sequences in the string, find your current position (start at 0) and the position of the next space. Then that's the word. Set the current position to the space and do it again. Keep repeating this until you're at the end.

By using the map you'll already have the word/count available.

If the job you're applying for requires university skills, I strongly recommend optimizing the map by adding some kind of hashing function. However, judging by the difficulty of the question I assume that that is not the case.

like image 35
Tom van der Woerdt Avatar answered Oct 17 '22 01:10

Tom van der Woerdt