ToExpression is the general Mathematica string-to-expression converter. If necessary you can write additional converters for the date fields, map "false" to False etc. We then perform some simple conversions (integers, empty fields, and general C-language numbers). These two operations are actually pretty fast and take about 1 sec.
Datagrip import csv code#
Next, the code splits the lines into separate fields at the commas. What it does first is removing all the quotes in your data (you don't need to do this if you don't want it, but I don't think you need quotes in the strings). It is possible to do your own custom conversions after reading in the raw strings in data: (data4 = (StringSplit & (StringReplace & data)) /.Ī_String?(StringMatchQ &) :> FromDigits,Ī_String?(StringMatchQ &) :> -FromDigits], While working, SemanticImport also needed about twice the space that Import needed. Memory usage is now a whopping 2 GB because it returns a Dataset to represent the input, which is very inefficient for flat data. Of course, there's SemanticImport that has more sophisticated methods to determine field types, but in practice it is hardly usable: (data3 = SemanticImport ) //AbsoluteTiming//First Memory usage is much larger now: data2 // ByteCountĪnd Import also needed about 1.2 GB during this process.
I guess your full, 1GB file will take 20 minutes or so to load. Plain importing as CSV does that and that's why it takes that long: (data2 = Import ) //AbsoluteTiming//First Each and every field has to be tested and converted separately. Hence, tesseract's solution won't work for you. It is impossible to specify a general format mask as needed by ReadList. Missing numerical data is marked by the string "NA". For instance, column 405 contains both "CHIEF EXECUTIVE OFFICER" and -1. The main issue is that your columns contain mixed data. It's only slightly larger than the file on disk: FileByteCount
Memory use is modest too: data // ByteCount (data = ReadList) // AbsoluteTiming//First I can read the data as strings quite fast.
0 kommentar(er)
