Extract only words from a cell array in matlab

Question

I have a set of documents containing pre-processed texts from html pages. They are already given to me. I want to extract only the words from it. I do not want any numbers or common words or any single letters to be extracted. The first problem I am facing is this.

Suppose I have a cell array :

{'!' '!!' '!!!!)'  '!!!!thanks' '!!dogsbreath'    '!)'    '!--[endif]--'    '!--[if'}

I want to make the cell array having only the words - like this.

{'!!!!thanks' '!!dogsbreath' '!--[endif]--' '!--[if'}

And then convert this to this cell array

{'thanks' 'dogsbreath' 'endif' 'if'}

Is there any way to do this ?

---

Updated Requirement : Thanks to all of your answers. However I am facing a problem ! Let me illustrate this (Please note that the cell values are extracted text from HTML documents and hence may contain non ASCII values) -

{'!/bin/bash'    '![endif]'    '!take-a-long'    '!–photo'}

This gives me the answer

{'bin'    'bash'    'endif'    'take'    'a'    'long'    'â'    'photo' }

My Questions:

• Why is bin/bash and take-a-long being separated into three cells ? Its not a problem for me but still why? Can this be avoided. I mean all words coming from a single cell being combined into one.
• Notice that in '!–photo' there exists an non-ascii character â which esentially means a. Can a step be incorporated such that this transformation is automatic?
• I noticed that the text "it? __________ About the Author:" gives me "__________" as a word. Why is this so ?
• Also the text "2. areoplane 3. cactus 4. a_rinny_boo... 5. trumpet 6. window 7. curtain ... 173. gypsy_wagon..." returns a word as 'areoplane' 'cactus' 'a_rinny_boo' 'trumpet' 'window' 'curtain' 'gypsy_wagon'. I want the words 'a_rinny_boo' and ''gypsy_wagon to be 'a' 'rinny' 'boo' 'gypsy' 'wagon'. Can this be done ?

---

Update 1 Following all the suggestions I have written down a function which does most of the things except the above two newly asked questions.

function [Text_Data] = raw_txt_gn(filename)

% This function will convert the text documnets into raw text
% It will remove all commas empty cells and other special characters
% It will also convert all the words of the text documents into lowercase

T = textread(filename, '%s');

% find all the important indices
ind1=find(ismember(T,':WebpageTitle:'));
T1 = T(ind1+1:end,1);

% Remove things which are not basically words
not_words = {'##','-',':ImageSurroundingText:',':WebpageDescription:',':WebpageKeywords:',' '};

T2 = []; count = 1;
for j=1:length(T1)    
    x = T1{j};
    ind=find(ismember(not_words,x), 1);
    if isempty(ind)

        B = regexp(x, '\w*', 'match');
        B(cellfun('isempty', B)) = []; % Clean out empty cells
        B = [B{:}]; % Flatten cell array

        % convert the string into lowecase
        % so that while generating the features the case sensitivity is
        % handled well
        x = lower(B);        

        T2{count,1} = x;
        count = count+1;
    end
end
T2 = T2(~cellfun('isempty',T2));


% Getting the common words in the english language
% found from Wikipedia
not_words2 = {'the','be','to','of','and','a','in','that','have','i'};
not_words2 = [not_words2, 'it' 'for' 'not' 'on' 'with' 'he' 'as' 'you' 'do' 'at'];
not_words2 = [not_words2, 'this' 'but' 'his' 'by' 'from' 'they' 'we' 'say' 'her' 'she'];
not_words2 = [not_words2, 'or' 'an' 'will' 'my' 'one' 'all' 'would' 'there' 'their' 'what'];
not_words2 = [not_words2, 'so' 'up' 'out' 'if' 'about' 'who' 'get' 'which' 'go' 'me'];
not_words2 = [not_words2, 'when' 'make' 'can' 'like' 'time' 'no' 'just' 'him' 'know' 'take'];
not_words2 = [not_words2, 'people' 'into' 'year' 'your' 'good' 'some' 'could' 'them' 'see' 'other'];
not_words2 = [not_words2, 'than' 'then' 'now' 'look' 'only' 'come' 'its' 'over' 'think' 'also'];
not_words2 = [not_words2, 'back' 'after' 'use' 'two' 'how' 'our' 'work' 'first' 'well' 'way'];
not_words2 = [not_words2, 'even' 'new' 'want' 'because' 'any' 'these' 'give' 'day' 'most' 'us'];

for j=1:length(T2)
    x = T2{j};
    % if a particular cell contains only numbers then make it empty
    if sum(isstrprop(x, 'digit'))~=0
        T2{j} = [];
    end
    % also remove single character cells
    if length(x)==1
        T2{j} = [];
    end
    % also remove the most common words from the dictionary
    % the common words are taken from the english dicitonary (source
    % wikipedia)
    ind=find(ismember(not_words2,x), 1);
    if isempty(ind)==0
        T2{j} = [];
    end
end

Text_Data = T2(~cellfun('isempty',T2));

---

Update 2 I found this code in here which tells me how to check for non-ascii characters. Incorporating this code snippet in Matlab as

% remove the non-ascii characters
if all(x  < 128)
else
  T2{j} = [];
end

and then removing the empty cells it seems my second requirement is fulfilled though the text containing a part of non-ascii characters completely disappears.

---

Can my final requirements be completed ? Most of them concerns the character '_' and '-'.

Answer 1

A regexp approach to go directly to the final step:

A = {'!' '!!' '!!!!)'  '!!!!thanks' '!!dogsbreath'    '!)'    '!--[endif]--'    '!--[if'};

B = regexp(A, '\w*', 'match');
B(cellfun('isempty', B)) = []; % Clean out empty cells
B = [B{:}]; % Flatten cell array

Which matches any alphabetic, numeric, or underscore character. For the sample case we get a 1x4 cell array:

B = 

    'thanks'    'dogsbreath'    'endif'    'if'

---

Edit:

Why is bin/bash and take-a-long being separated into three cells ? Its not a problem for me but still why? Can this be avoided. I mean all words coming from a single cell being combined into one.

Because I'm flattening the cell arrays to remove nested cells. If you remove B = [B{:}]; each cell will have a nested cell inside containing all of the matches for the input cell array. You can combine these however you want after.

Notice that in '!–photo' there exists an non-ascii character â which esentially means a. Can a step be incorporated such that this transformation is automatic?

Yes, you'll have to make it based on the character codes.

I noticed that the text "it? __________ About the Author:" gives me "__________" as a word. Why is this so ?

As I said, the regex matches alphabetic, numeric, or underscore characters. You can change your filter to exclude _, which will also address the fourth bullet point: B = regexp(A, '[a-zA-Z0-9]*', 'match'); This will match a-z, A-Z, and 0-9 only. This will also exclude the non-ASCII characters, which it seems like the \w* flag matches.