Camspiers\StatisticalClassifier\Normalizer\Token\NormalizerInterface::normalize PHP Method

normalize() public method

This is to increase matches across what tokens are deemed equivalent but differnt
public normalize ( array $tokens ) : array
$tokens array The tokens to normalizer
return array The array of normalized tokens
    public function normalize(array $tokens);

Usage Example

 public function __invoke($data)
 {
     foreach ($data as $category => $documents) {
         foreach ($documents as $index => $document) {
             if ($this->documentNormalizer) {
                 $document = $this->documentNormalizer->normalize($document);
             }
             $tokens = $this->tokenizer->tokenize($document);
             if ($this->tokenNormalizer) {
                 $tokens = $this->tokenNormalizer->normalize($tokens);
             }
             $data[$category][$index] = $tokens;
         }
     }
     return $data;
 }
All Usage Examples Of Camspiers\StatisticalClassifier\Normalizer\Token\NormalizerInterface::normalize
NormalizerInterface