[SOLVED] Software to handle l o t s of data (reports *.CSV)

dockland · 2016-05-30 17:58:48

At work I use Excel 2016 64-bit getting and managing reports from our service desk. The amount of data is huge. Hundreds of thousands fields, approx 60 columns * 8000 rows a month with different amount of data and information.
The export file is *.csv or *.xls and I tried to use Libreoffice 5 to handle it. It constantly crashes trying to do some pivot tables based on the information.

Is there any other application that can handle huge amount of data in a simple quick way to get the statistics? With bars, and easy to get explanational summaries out in a readable bars or graphs? I need to run this report once a month and i can offer one hour to get it "understandable" to our management.

Last edited by dockland (2016-05-30 19:44:01)

ngoonee · 2016-05-30 19:05:22

If libreoffice and gnumeric can't handle it you could either try to roll your own solution (assembly required) with python and pandas for example, or consider a cloud solution (office 365 or google sheets).

dcdriving · 2016-05-30 19:27:59

I'm sure libreoffice base should handle that amount of data but importing it would mean copying and pasting which might cause a problem too. The other way around it would be to open the .csv file in calc, save it as a dbase file and then open the dbase file in base. This would then directly edit the dbase file I think? This would also then allow you to manipulate the data as a table with sql and perhaps break it down a bit to export more manageable chunks to individual calc sheets for graphs. You would need to ensure that the first row of the csv file contained row headings though.

Trilby · 2016-05-30 19:32:58

8000 rows by ~60 columns doesn't even get remotely close to "big data". But that said, anyone who does work with "big data" wouldn't dream of opening it in excel. I do agree with the above, that data of this size should be manageable in excel or LO's spreadsheet thingy ... but if you want something efficient, Use R.

And if you want to automate nice report generation, use Sweave (R + latex).

For simple descriptive plots and/or bar graphs, this might be a dozen or so lines of R code. With a Sweave document you'd basically hit "go" once a month and your new report would be regenerated in seconds.

dockland · 2016-05-30 19:43:37

Trilby wrote:

8000 rows by ~60 columns doesn't even get remotely close to "big data". But that said, anyone who does work with "big data" wouldn't dream of opening it in excel. I do agree with the above, that data of this size should be manageable in excel or LO's spreadsheet thingy ... but if you want something efficient, Use R.
And if you want to automate nice report generation, use Sweave (R + latex).
For simple descriptive plots and/or bar graphs, this might be a dozen or so lines of R code. With a Sweave document you'd basically hit "go" once a month and your new report would be regenerated in seconds.

Thank you all. I know it's not "big" data, but for me it is Thanks all. Will try Sweave (and try out Google Fusiontables aswell.

Arch Linux

#1 2016-05-30 17:58:48

[SOLVED] Software to handle l o t s of data (reports *.CSV)

#2 2016-05-30 19:05:22

Re: [SOLVED] Software to handle l o t s of data (reports *.CSV)

#3 2016-05-30 19:27:59

Re: [SOLVED] Software to handle l o t s of data (reports *.CSV)

#4 2016-05-30 19:32:58

Re: [SOLVED] Software to handle l o t s of data (reports *.CSV)

#5 2016-05-30 19:43:37

Re: [SOLVED] Software to handle l o t s of data (reports *.CSV)

Board footer