I'm doing my research project now with some quite large amount of data. I'm not a programmer and I my knowledge is limited to Excel and LibreOffice Calc, when we talk about preparation of the data for the analysis.

My problem is with LibreOffice Calc, where my data has ~500 000 of lines. MS excell just crunches it with no sweat (insert new column, do some vlookup's). However LibreOffice Calc just stalks when I ask it to insert new column or even sort that data. And stalks it from 10 to 60 or more seconds. I went to tools/options/memory and increased some numbers there, but no positive effect registered.
My Linux machine, on which I work, has no problems with resources: 8GB RAM, intel Core i7 CPU, SSD drive.
My Windows machine, where excel has no any problems with that large amount of data has even even worse CPU (i5).

I have also noticed, that during these stalks only one CPU core reaches 100 percent, all other has no load on my Linux machine.

I guess something could be tweaked. Anyone can suggest anything?

P.S. My project results looks like that:
https://flic.kr/p/USpV3C
https://flic.kr/p/TB8BZY
It is Lithuanian Parliament votes rendered with Gephi.

Might be worth asking over at https://ask.libreoffice.org/en/questions/

Thanks. I'll do that.

It seems like there is quite a lot of similar bug reports for poor calc performance with large data. It seems like this is not a configuration issue. Or not the configuration issue, which I could solve... I'll wait for next LibreOffice version.

Thanks for the link again anyways!

1 members found this post helpful.

I tried to use Calc against updating MS excel but the speed on some large calculations made buying the MS product cost effective. Excel rates well for many tasks. Strictly on a DB issue there are other programs that may be faster but you'll have to learn them. IBM DB2 has a host of accelerator products for massive databases.

heres my quick-and-dirty research:
http://www.linuxquestions.org/questi...7/#post5216232

Yeah... Common problem. But it seems like the best solution is to use right tool. And either Calc or Excel is not the right tool for large amount of data. I have to learn to use another tool. Are there any open source tools for that? I guess IBM DB2 is not an Open Source. Or is it?

Depending on what you are doing, R might be an option. It is very widely used.

ibm-db2 isnt (it runs on s/390 mainframes -- the version of db2 for unix would be ibm-udb which also is proprietary) but mysql is.

Just a thought.

How does gNumeric stack up against MsExcel? As a quck hit would anyone recommend that the OP tries it before moving away from spreadsheets?

OK

It boils down to the type of data and the way you want to access it and crunch it really.
You'd have to decide what tool is best.

Official differences. https://wiki.openoffice.org/wiki/Doc...Calc_and_Excel One can generally export/import data of these or link their data to some standard database to speed up or increase ability as in size or scope or join. You can use different backends with either calc or excel I think.

From OODB this is about right. https://en.wikipedia.org/wiki/Compar...gement_systems

Some data base here. http://www.yolinux.com/TUTORIALS/LinuxDatabases.html

Ideas. http://wiki.linuxquestions.org/wiki/...ndows_software

Then you get into similar things like Filemaker pro and ways to use similar in linux.

If one had a huge amount of data to crunch in some way and they needed it fast, IBM DB2 and maybe Oracle would be my first two choices. My huge and your huge may be two different things.

Thanks for the links. I'll try to understand what are these.

My large amount of data is ~5.5 mil of rows. It's not a secret (all recorded voting from Lithuanian Parliament (from http://lrs.lt)): http://atviriduomenys.lt/data/lrs/balsavimai/

I'm trying to analyze it using Gephi. I render these results into coalitions: https://flic.kr/p/TB8BZY . It seems like fun to me. It also helps me to understand Lithuanian politics better and quite faster than it would be otherwise.

And functions, which I use with the excel or calc is: Concatenate, Vlookup, filter.

Nice graphic. It seems as though flickr thinks it's a firework.

I'm curious about how you are parsing those files. The header in "klausimai.csv" shows six fields but the number of commas in a record varies. For example the following line has 40 commas.Kind of hard to import as a csv file.

EDIT: Never mind. I was trying to import some of the data into MySQL and it was ignoring the quotes around the "klausimai.pavadinimas" field. Figured it out by importing into LibreOffice calc.

Please elaborate on how you manipulate the data.

Here is one of the "Gephi-ready" files: https://1drv.ms/f/s!Alz2FMnmDud9huNSn2YTASmvCoEGpw

Here's Gephi file itself: https://1drv.ms/f/s!Alz2FMnmDud9huNTSppCmXljMBve2Q

All politicians has their fractions and then coalitions. And some of them changes fractions during the ruling period - I needed to put all these fractions next to the name. I then colored all these fractions, so I could see how these renders into the coalitions.