{"id":227,"date":"2016-01-05T16:02:58","date_gmt":"2016-01-05T16:02:58","guid":{"rendered":"http:\/\/o-calcpro.com\/wiki\/?p=227"},"modified":"2016-01-05T16:03:52","modified_gmt":"2016-01-05T16:03:52","slug":"fast-indexing-of-pplx-data","status":"publish","type":"post","link":"http:\/\/o-calcpro.com\/wiki\/2016\/01\/05\/fast-indexing-of-pplx-data\/","title":{"rendered":"Fast indexing of PPLX data"},"content":{"rendered":"

One of the tasks that sometimes comes up during plugin development is the need to rapidly parse and spatially index a large number of PPLX files.<\/p>\n

Performing this task by fully hydrating each PPLX file into O-Calc proper, while relatively quick, would still take more time than desired for a large number of poles. In this post is presented a code snippet that can parse the latitude, longitude, poleID, and MCU percentage from a PPLX file very rapidly and allow for the processing of tens to hundreds of PPLX files per second depending on your system’s IO performance.<\/p>\n

First I need to create a structure to hold the data I intend to parse form the PPLX and then instantiate a list to hold an instance of that structure for each PPLX that I parse.<\/p>\n

\n\n\t\t
\n\n\t\t
\n\t\t\t\t\t\t< View plain text<\/span> ><\/a>\n\t\t\t\t\t<\/div>\n\n\t\t
C#<\/div>\n\n\t\t\n\t\t
\n\n\t<\/div>\n\t\n\t
\n\t\t
  1. public<\/span> struct<\/span> Pole<\/div><\/li>\n
  2. {<\/span><\/div><\/li>\n
  3.     public<\/span> double<\/span> cLat;<\/span><\/div><\/li>\n
  4.     public<\/span> double<\/span> cLon;<\/span><\/div><\/li>\n
  5.     public<\/span> string<\/span> cName;<\/span><\/div><\/li>\n
  6.     public<\/span> string<\/span> cPath;<\/span><\/div><\/li>\n
  7.     public<\/span> double<\/span> cMCU;<\/span><\/div><\/li>\n
  8. }<\/span><\/div><\/li>\n
  9.  <\/div><\/li>\n
  10. public<\/span> List<<\/span>Pole><\/span> cPoles =<\/span> new<\/span> List<<\/span>Pole><\/span>(<\/span>)<\/span>;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n

    Now if I create a list of file path strings of all of the PPLX files I wish to process *the construction of this list is not covered here but typically would involce one or more calls to files = Directory.GetFiles(directory, “*.pplx”);) I can process them as follows:<\/p>\n

    \n\n\t\t
    \n\n\t\t
    \n\t\t\t\t\t\t< View plain text<\/span> ><\/a>\n\t\t\t\t\t<\/div>\n\n\t\t
    C#<\/div>\n\n\t\t\n\t\t
    \n\n\t<\/div>\n\t\n\t
    \n\t\t
    1. foreach<\/span> (<\/span>String<\/span> file in<\/span> files)<\/span><\/div><\/li>\n
    2. {<\/span><\/div><\/li>\n
    3.     using<\/span> (<\/span>StreamReader sr =<\/span> new<\/span> StreamReader(<\/span>file)<\/span>)<\/span><\/div><\/li>\n
    4.     {<\/span><\/div><\/li>\n
    5.         using<\/span> (<\/span>XmlTextReader xr =<\/span> new<\/span> System.Xml<\/span><\/span>.<\/span>XmlTextReader<\/span>(<\/span>sr)<\/span>)<\/span><\/div><\/li>\n
    6.         {<\/span><\/div><\/li>\n
    7.             String<\/span> poleId =<\/span> System.IO<\/span><\/span>.<\/span>Path<\/span>.<\/span>GetFileNameWithoutExtension<\/span>(<\/span>file)<\/span>;<\/span><\/div><\/li>\n
    8.             double<\/span> lat =<\/span> 0<\/span>;<\/span><\/div><\/li>\n
    9.             double<\/span> lon =<\/span> 0<\/span>;<\/span><\/div><\/li>\n
    10.             while<\/span> (<\/span>xr.<\/span>Read<\/span>(<\/span>)<\/span>)<\/span><\/div><\/li>\n
    11.             {<\/span><\/div><\/li>\n
    12.                 if<\/span> (<\/span>xr.<\/span>NodeType<\/span> ==<\/span> System.Xml<\/span><\/span>.<\/span>XmlNodeType<\/span>.<\/span>Element<\/span> &&<\/span> xr.<\/span>Name<\/span> ==<\/span> "VALUE"<\/span>)<\/span><\/div><\/li>\n
    13.                 {<\/span><\/div><\/li>\n
    14.                     String<\/span> sname =<\/span> xr.<\/span>GetAttribute<\/span>(<\/span>"NAME"<\/span>)<\/span>;<\/span><\/div><\/li>\n
    15.                     if<\/span> (<\/span>sname ==<\/span> "Latitude"<\/span>)<\/span><\/div><\/li>\n
    16.                     {<\/span><\/div><\/li>\n
    17.                         xr.<\/span>Read<\/span>(<\/span>)<\/span>;<\/span><\/div><\/li>\n
    18.                         if<\/span> (<\/span>!<\/span>double<\/span>.<\/span>TryParse<\/span>(<\/span>xr.<\/span>Value<\/span>, out<\/span> lat)<\/span>)<\/span> break<\/span>;<\/span><\/div><\/li>\n
    19.                         if<\/span> (<\/span>lat ==<\/span> 0<\/span>)<\/span> break<\/span>;<\/span><\/div><\/li>\n
    20.                     }<\/span><\/div><\/li>\n
    21.                     else<\/span> if<\/span> (<\/span>sname ==<\/span> "Longitude"<\/span>)<\/span><\/div><\/li>\n
    22.                     {<\/span><\/div><\/li>\n
    23.                         xr.<\/span>Read<\/span>(<\/span>)<\/span>;<\/span><\/div><\/li>\n
    24.                         if<\/span> (<\/span>!<\/span>double<\/span>.<\/span>TryParse<\/span>(<\/span>xr.<\/span>Value<\/span>, out<\/span> lon)<\/span>)<\/span> break<\/span>;<\/span><\/div><\/li>\n
    25.                         if<\/span> (<\/span>lon ==<\/span> 0<\/span>)<\/span> break<\/span>;<\/span><\/div><\/li>\n
    26.  <\/div><\/li>\n
    27.                         double<\/span> mcu =<\/span> -<\/span>1<\/span>;<\/span><\/div><\/li>\n
    28.                         try<\/span><\/div><\/li>\n
    29.                         {<\/span><\/div><\/li>\n
    30.                             sr.<\/span>DiscardBufferedData<\/span>(<\/span>)<\/span>;<\/span><\/div><\/li>\n
    31.                             sr.<\/span>BaseStream<\/span>.<\/span>Seek<\/span>(<\/span>-<\/span>1024<\/span>, SeekOrigin.<\/span>End<\/span>)<\/span>;<\/span><\/div><\/li>\n
    32.                             String<\/span> sbuff =<\/span> sr.<\/span>ReadToEnd<\/span>(<\/span>)<\/span>;<\/span><\/div><\/li>\n
    33.                             int<\/span> idx =<\/span> sbuff.<\/span>IndexOf<\/span>(<\/span>"PercentAtMCU"<\/span>)<\/span>;<\/span><\/div><\/li>\n
    34.                             if<\/span> (<\/span>idx ><\/span> 0<\/span>)<\/span><\/div><\/li>\n
    35.                             {<\/span><\/div><\/li>\n
    36.                                 mcu =<\/span> Parse(<\/span>sbuff.<\/span>Substring<\/span>(<\/span>idx +<\/span> 20<\/span>, 100<\/span>)<\/span>)<\/span>;<\/span><\/div><\/li>\n
    37.                             }<\/span><\/div><\/li>\n
    38.                         }<\/span><\/div><\/li>\n
    39.                         catch<\/span><\/div><\/li>\n
    40.                         {<\/span><\/div><\/li>\n
    41.                             mcu =<\/span> -<\/span>1<\/span>;<\/span><\/div><\/li>\n
    42.                         }<\/span><\/div><\/li>\n
    43.  <\/div><\/li>\n
    44.                         if<\/span> (<\/span>lat !=<\/span> 0<\/span> &&<\/span> lon !=<\/span> 0<\/span>)<\/span><\/div><\/li>\n
    45.                         {<\/span><\/div><\/li>\n
    46.                             Pole pole =<\/span> new<\/span> Pole(<\/span>)<\/span>;<\/span><\/div><\/li>\n
    47.                             pole.<\/span>cLat<\/span> =<\/span> lat;<\/span><\/div><\/li>\n
    48.                             pole.<\/span>cLon<\/span> =<\/span> lon;<\/span><\/div><\/li>\n
    49.                             pole.<\/span>cName<\/span> =<\/span> poleId;<\/span><\/div><\/li>\n
    50.                             pole.<\/span>cMCU<\/span> =<\/span> mcu;<\/span><\/div><\/li>\n
    51.                             pole.<\/span>cPath<\/span> =<\/span> file;<\/span><\/div><\/li>\n
    52.                             cPoles.<\/span>Add<\/span>(<\/span>pole)<\/span>;<\/span><\/div><\/li>\n
    53.                         }<\/span><\/div><\/li>\n
    54.                         break<\/span>;<\/span><\/div><\/li>\n
    55.  <\/div><\/li>\n
    56.                     }<\/span><\/div><\/li>\n
    57.                 }<\/span><\/div><\/li>\n
    58.             }<\/span><\/div><\/li>\n
    59.         }<\/span><\/div><\/li>\n
    60.     }<\/span><\/div><\/li>\n
    61. }<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n

      Supported by…<\/p>\n

      \n\n\t\t
      \n\n\t\t
      \n\t\t\t\t\t\t< View plain text<\/span> ><\/a>\n\t\t\t\t\t<\/div>\n\n\t\t
      C#<\/div>\n\n\t\t\n\t\t
      \n\n\t<\/div>\n\t\n\t
      \n\t\t
      1. double<\/span> Parse(<\/span>String<\/span> pStr)<\/span><\/div><\/li>\n
      2. {<\/span><\/div><\/li>\n
      3.     const<\/span> char<\/span> gt =<\/span> (<\/span>char<\/span>)<\/span>62<\/span>;<\/span><\/div><\/li>\n
      4.     const<\/span> char<\/span> lt =<\/span> (<\/span>char<\/span>)<\/span>60<\/span>;<\/span><\/div><\/li>\n
      5.     int<\/span> idx =<\/span> pStr.<\/span>IndexOf<\/span>(<\/span>gt)<\/span>;<\/span><\/div><\/li>\n
      6.     String<\/span> s =<\/span> pStr.<\/span>Substring<\/span>(<\/span>idx +<\/span> 1<\/span>)<\/span>;<\/span><\/div><\/li>\n
      7.     idx =<\/span> s.<\/span>IndexOf<\/span>(<\/span>lt)<\/span>;<\/span><\/div><\/li>\n
      8.     s =<\/span> s.<\/span>Substring<\/span>(<\/span>0<\/span>, idx)<\/span>;<\/span><\/div><\/li>\n
      9.     double<\/span> v =<\/span> 0<\/span>;<\/span><\/div><\/li>\n
      10.     if<\/span>(<\/span>double<\/span>.<\/span>TryParse<\/span>(<\/span>s, out<\/span> v)<\/span>)<\/span> return<\/span> v;<\/span><\/div><\/li>\n
      11.     return<\/span> 0<\/span>;<\/span><\/div><\/li>\n
      12. }<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n

        The following video shows this code in operation…<\/p>\n