Reading and writing binary files in R & Fortran

ASCII files (i.e., human readable text files) are great for storing small amounts of data, because you can always retrieve the data without great difficulty. But they become impractical for large amounts of data. In that case you might want to consider saving data in binary format. (R is discussed first, Fortran second.) This has numerous advantages:

• Most importantly is precision---storing data in ASCII files usually results in precision loss; binary files store the information exactly as represented in the computers memory.
• A further advantage is that in general, binary files require less disk space. This is because in ASCII files each value is stored as a sequence of characters, and each character requires a couple of bytes storage, whereas in binary format each value is stored in a fixed small amount of bytes.
• Furthermore, data can be quickly retrieved from a binary file. This is because they don't have to be read and parsed into a value as ASCII representations have to be---this is absolutely essential when processing huge amounts of data that cannot be stored in the computers main memory.
• In fact, the latter necessitates the ability to adress only parts of the data in a file, and this is close to impossible with ASCII files, but is possible in binary files with the seek function.

In R, reading and writing binary files is essentially as simple as reading and writing from and to ASCII files. However, one needs to keep track of the type of data that have been stored in a file, because in binary files data are stored simply as binary patterns (who would have thought of that!?), and the interpretation of these patterns totally depends on the type (i.e. integer, double, character, etc.) they were assigned. In addition it requires some bookkeeping if you want to access the data in arbitrary order ('random access'). Here's how:

## basic reading and writing
con = file(file.choose(), open="wb")   # opens binary file to write to
writeBin(rnorm(100),con)               # stores data in binary format
close(con)                             # closes the file
con = file(file.choose(), open="rb")   # opens binary file for reading
readBin(con, double(), n=100)          # reads 100 values of type double

## simultaneous reading and writing, and seek
con = file(file.choose(), open="r+b")  # open file to read and write
oldpos = seek(con, where=8*3))         # moves to position after 3rd 
                                       #    double (=8 bytes)
readBin(con,double(),n=1)              # reads 1 value
seek(con, where=oldpos)                # back to old position
readBin(con,double(),n=4)              # last value equals the above
oldpos = seek(con, where=8*4, rw='write')
                                       # go to 4th double to write 
                                       #    there
writeBin(pi,con)                       # store pi at current position
seek(con, where=oldpos)                # back to old position
readBin(con,double(),n=4)              # last number now equals pi

R also has the capability to read and write compressed files---see help('file')

A similar thing can be done in Fortran 90 (Conor needed this) and is achieved as follows (this is, by the way, not very efficient in terms of disk space):

! variables
  integer        :: do_nrchan = 151, do_nrsamp = 128
  double complex :: value(do_nrchan,do_nrsamp), grbge
  integer        :: do_sysfile
  integer        :: reclength, recnr, trialnr = 13

  trialnr = 5
! open file for binary IO (input/output)
  inquire(iolength=reclength) grbge, value   
  reclength = reclength*4
    ! double complex == 16bytes,
    ! reclength now equals #bytes( value & grbge )
  open(unit=do_sysfile, file=do_sysfilename,  &
       access='direct', recl=reclength,  &
       form='binary', err=9, status=stat)

! write
  recnr = trialnr
  write( do_sysfile, REC=recnr, ERR=7 ) grbge, value

! read
  recnr = trialnr
  read( do_sysfile, rec=recnr, err=7 ) grbge, value
  .
  .
  .
  end
7 print *, "read error"
  stop
9 print *, "write error"
  stop