Re: How to efficiently read binary files?

Grant Edwards wrote:
> On 2006-04-30, David Lees <debl2NoSpam@verizon.net> wrote:
>
>>I want to process large binary files (>2GB) in Python. I have played
>>around with prototypes in pure Python and profiled the code. Most of
>>the time seems to be spent converting back and forth to and from strings
>>using the struct module. Is there a way to directly read into an array
>>of integers in Python?
>
> Perhaps the numarray module?

numpy for new code, please. In particular, numarray is limited by 32-bit APIs
even on 64-bit platforms, so a >2GB file will be difficult to process even when
using mmap. numpy removed this restriction on 64-bit platforms. 32-bit users
will still have to split up the file into <2GB chunks, though.

  http://numeric.scipy.org/
  https://lists.sourceforge.net/lists/listinfo/numpy-discussion

-- 
Robert Kern
robert.kern@gmail.com
"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco