[Bast-commits] r5231 - in SQL-Abstract/1.x/branches/1.50_RC-extraparens: . lib/SQL/Abstract

[norbi at dev.catalyst.perl.org]

Author: norbi
Date: 2008-12-10 10:06:24 +0000 (Wed, 10 Dec 2008)
New Revision: 5231

Modified:
   SQL-Abstract/1.x/branches/1.50_RC-extraparens/
   SQL-Abstract/1.x/branches/1.50_RC-extraparens/lib/SQL/Abstract/Test.pm
Log:
 r5301 at vger:  mendel | 2008-12-10 11:06:13 +0100
  * Rewritten tokenizer to use foreach instead of map and made the regex more efficient.



Property changes on: SQL-Abstract/1.x/branches/1.50_RC-extraparens
___________________________________________________________________
Name: svk:merge
   - 4d5fae46-8e6a-4e08-abee-817e9fb894a2:/local/bast/SQL-Abstract/1.x/branches/1.50_RC:5202
4d5fae46-8e6a-4e08-abee-817e9fb894a2:/local/bast/SQL-Abstract/1.x/branches/1.50_RC-extraparens:5269
   + 4d5fae46-8e6a-4e08-abee-817e9fb894a2:/local/bast/SQL-Abstract/1.x/branches/1.50_RC:5202
4d5fae46-8e6a-4e08-abee-817e9fb894a2:/local/bast/SQL-Abstract/1.x/branches/1.50_RC-extraparens:5301

Modified: SQL-Abstract/1.x/branches/1.50_RC-extraparens/lib/SQL/Abstract/Test.pm
===================================================================
--- SQL-Abstract/1.x/branches/1.50_RC-extraparens/lib/SQL/Abstract/Test.pm	2008-12-10 01:28:10 UTC (rev 5230)
+++ SQL-Abstract/1.x/branches/1.50_RC-extraparens/lib/SQL/Abstract/Test.pm	2008-12-10 10:06:24 UTC (rev 5231)
@@ -163,11 +163,13 @@
   my $s = shift;
 
   # tokenize string, and remove all optional whitespace
-  my $tokens = [
-    grep {!/^$/}
-      map { s/\s+/ /g; s/\s*([^\w\s]+)\s*/$1/g; $_ }
-        split $tokenizer_re, $s
-  ];
+  my $tokens = [];
+  foreach my $token (split $tokenizer_re, $s) {
+    $token =~ s/\s+/ /g;
+    $token =~ s/\s+([^\w\s])/$1/g;
+    $token =~ s/([^\w\s])\s+/$1/g;
+    push @$tokens, $token if $token !~ /^$/;
+  }
 
   my $tree = _recurse_parse($tokens, PARSE_TOP_LEVEL);
   return $tree;