Working with UNIX Programming Tools

1. UNIX Design Philosophy
2. Compiling Programs
   1. Using FORTRAN
      • Using Multiple Source Files
      • Considerations for Porting Existing FORTRAN Code
      • SAVE Statement May Be Required
      • Specifying File Definitions for Input/Output Units
   2. Using the C Programming Language
3. Redirecting Standard Error
   1. Bourne and Korn Shells
   2. C Shell
4. Background Processing
   1. Background Jobs and Logout -- the "nohup" command
      • Bourne and Korn Shells
      • C Shell
   2. Changing Priority: nice
   3. Monitoring Process Status
      • Using the tail Command to Monitor Progress of a Background Job
   4. Terminating a Process
   5. Obtaining Execution Times
      • Bourne and Korn Shells
      • C Shell
   6. Using the Profiler to Determine Time Spent in SubProgram Components
5. UNIX Program Development Tools
   1. Using the make Utility
   2. Using the Archive Librarian
6. Mixed Language Programming: Timing Portions of FORTRAN Code Using a C Language Function

UNIX has become popular among program developers as it includes several features which they find to be very desirable:

• Portability (Machine Independence)
• A Powerful Multi-User System
  • Shell Command Interpreter
  • Hierarchical File Structure
  • Multi-Tasking with Background Processing Capabilities (&)
  • Extensible via the C Language
• Networking Capabilities
• Security Features
• Widely Accepted

UNIX Design Philosophy

To work with greatest effectiveness in the UNIX programming environment, you should be aware of some of the design characteristics of the operating system:

• Simplicity
  • Short commands: minimal typing
  • "No News is Good News!"
• Modularity: Write programs so that their output can be used as input by other programs.
• Power and Functionality: Provide a wealth of single purpose utilities.
• Confidence: You know what you are doing!
  • If you erase a file, it is gone.
  • If you use the name of an existing file as the target for a rename or copy, the original file is replaced by the new file.

Compiling Programs

Compilers are available on VTAIX for FORTRAN (xlf or f77 command), the C programming language (xlc or cc command), and Pascal (xlp command). Providing arguments are passed correctly, it is possible to write programs which use a mixture of these languages.

Using FORTRAN

xlf is IBM's "eXceptionaL Fortran" compiler which is an ANSI standard FORTRAN compiler with IBM extensions. If you are using a non-AIX version of UNIX, use the command "f77" in place of "xlf" in the examples which follow.

The FORTRAN compiler xlf on VTAIX is invoked by entering the command "xlf" or "f77" followed by compilation options and the names of the source files you wish to compile. FORTRAN source files must end in the characters ".f"; otherwise the compiler will reject the source file for ending in an invalid suffix. For example, to compile FORTRAN source code contained in the file "prog_name.f" and create the executable module "a.out", enter:

xlf  prog_name.f
	

 ./a.out
	

Use the "-o" option if you wish to specify the name of the output executable file to be created by the compilation. For example, to assign the name "my_prog" to the executable created by compiling prog_name.f, enter:

xlf -o my_prog prog_name.f
	

 ./my_prog
	

Additional options include:

-c

compile only (do not link)

-C

perform run-time checking of array bounds and character substring expressions

-g

produce debug information

-lname

search library "name"

-o

name the executable module (instead of a.out)

-O

perform optimization

-qsource

produce an output source listing

For additional information about the compiler and option flags, enter:

man  xlf
	

Using Multiple Source Files

You can compile and link more than one file when you invoke the FORTRAN complier. For example, to create an executable program called "statsall" from the subprogram modules "statsmain.f", "means.f", and "std.f" from within the xmp directory, enter:

xlf -o statsall statsmain.f means.f std.f
	

Considerations for Porting Existing FORTRAN Code

The xlf FORTRAN compiler on VTAIX is an ANSI FORTRAN 77 compiler with IBM extensions. If you limit yourself to using the FORTRAN 77 standard, you should be able to port code to and from xlf with minimal difficulty. Since xlf includes more extensions than VS FORTRAN, it is anticipated that individuals who use IBM extensions will encounter more problems porting xlf applications to other environments than will be encountered in porting applications to xlf.

The RS/6000 uses double precision arithmetic in its computations:

For most applications there is no reason to run Single Precision -- in some cases Single Precision Programs will run slower than Double as the machine does all of its Real computations in Double Precision and rounds to create Single Precision numbers.

xlf uses IEEE Floating Point Representation:

This may result in different precision and ranges for numbers, for example:

• The approximate non-zero minimum double precision number is 2.225074D-308
• The approximate absolute maximum double precision number is 1.797693D+308
• The approximate precision is 7 single precision (REAL*4) digits, 15 double precision (REAL*8), and 31 quad precision (REAL*16) digits.

xlf may treat floating point exceptions differently than other compilers. Some notable differences include:

• Division by zero: the result is set equal to "INF" and processing continues
• Division by infinity: the result is set equal to zero

SAVE Statement

The SAVE Statement may be required to preserve results when a subroutine or function is reused. The xlf FORTRAN compiler does not necessarily preserve values upon exit from subroutines. Thus it may be necessary to add SAVE statements in some or all of your subroutines to guarantee that values will be preserved from one invocation of the routine to the next. To provide maximum portability among systems, it is recommended that you use SAVE statements in your programs whenever you wish to preserve values from one subprogram call to the next.

Specifying File Definitions for Input/Output Units

In the UNIX environment, FORTRAN Unit 5 corresponds to standard input and Unit 6 corresponds to standard output. Redirection can be used in the standard UNIX fashion to associate files with these units.

Typically, OPEN statements are used within FORTRAN programs to associate external files with other unit numbers. By default, if no OPEN statement is used on VTAIX, output (or input) for other unit numbers will be written to (or read from) file fort.#, where # is a number corresponding to the unit used for output (or input), e.g., fort.1 will be used for unit 1 and fort.27 will be used for unit 27. If you are using another UNIX system, write output to units 1 and 27 and observe the filenames used as the defaults for these unit numbers.

If you prefer to use external commands to associate file names with unit numbers in your FORTRAN code, you can use the corresponding environmental variables or use an ln (link) command to assign these associations for you. The syntax required is compiler dependent and you will need to check its documentation to determine the syntax you should use.

For example, on VTAIX, you can use the following command to associate unit 10 with the file "my.data" in your "xyz" sub-directory:

ln  -s  ~/xyz/my.data  fort.10
	

Note: if the file fort.1 already exists, it should either be removed or moved to a new name prior to issuing the above symbolic link command.

Note: if you have already associated a file with a FORTRAN unit and would like to reassign the link, use the -fs option, e.g.:

ln  -fs  ~/xyz/my.data  fort.10
	

Using the C Programming Language

C programming language files which are to be compiled by the cc command must end in the characters ".c". On VTAIX, you can also invoke the eXceptionaL C compiler by using the xlc command.

The sample programs copied to your disk include several C source code files in the xmp/c subdirectory. Use the following command to compile 2cent.c which converts a value provided in standard input from Fahrenheit temperature to Centigrade and places the result in standard output:

    cc  2cent.c
	

You can execute the program by entering:

    ./a.out
	

The program waits until you enter a numeric value corresponding to a temperature in degrees Fahrenheit. The program will compute the corresponding value in degrees centigrade and display it on the screen. At this point you might be wondering why the program does not display a prompt to indicate the type of input which it requires. Previously, we had discussed the utility of filters, programs which accept standard input and write to standard output. Here is another example of a filter. This program can accept input piped from another application or pipe output to another program. We will illustrate use of this program as a filter in the exercises which follow.

In order to keep the files organized on your disk, let's rename the file a.out to 2cent using the mv (move) command:

    mv  a.out  2cent
	

As you have seen in the preceding example, cc assigns the default name "a.out" to the executable file it creates. To assign an alternative name to the executable when the source code is compiled, use the "-o" option of the compiler. For example, to compile 2fahren.c and assign the name 2fahren to the executable it creates, enter:

    cc  -o  2fahren  2fahren.c
	

Now we will use redirection to create two output files:

    ./2fahren > ctemp.data
	    53
	    ./2fahren > second.data
	    19.56
	

Observe that the value is no longer displayed on the screen.

Use the "cat" command to display the contents of the file "ctemp.data" and then use input redirection to use the stored data value and display the result on the screen:

    cat         ctemp.data
	    ./2cent < ctemp.data
	

Observe that the value displayed by the 2cent command is the same value you had originally input to the 2fahren command.

If you would like to add additional data to an output file, use the append (">>") redirection symbol:

    ./2fahren >> ctemp.data
	    29
	    cat          ctemp.data
	

Since both 2fahren and 2cent use standard input and standard output, they can be used as filters to process an input data value:

    ./2fahren < second.data | tee ctemp.data | ./2cent ;
	    12
	

The preceding command string uses converts the data value stored in second.data Fahrenheit to Centigrade, pipes the output to the "tee" command which places the result in the file ctemp.data and in standard output where it is piped to 2fahren. The result is the same as the value entered to the 2cent command. The converted Centigrade temperature is found in ctemp.data. Observe that the previously stored values in ctemp.data have been overwritten.

Note: File my_source.c contains a C program which prompts for an input value and then converts it from Fahrenheit to Centigrade.

You can compile and link more than one file when you invoke the C complier. For example, to create an executable program called "statsall" from the subprogram modules "statsmain.c", "means.c", and "std.c" from within the xmp/c directory, enter:

cc -o statsall statsmain.c means.c std.c
	

Redirecting Standard Error

Error diagnostics generated by a program are normally directed to the terminal screen even when the standard output has been redirected to a file; however, it is useful, at times, to be able to write the error diagnostics to a file. This section will show how these diagnostics can be written either to their own file, or to the same file as standard output.

Bourne and Korn Shells

Recall that you can use the redirections symbol (>) to place program output in a file:

   ./filesize  >  output
	   cat  output
	

In a similar fashion you can redirect standard error by preceding the redirection symbol by the number "2":

 ./cars   2>   err
	 cat   err
	

When redirecting both standard output and standard error, it is customary to precede the standard output redirection by the number "1":

 ./my_prog  1> output  2> err
	

To send both standard output and standard error to the same file, use &1 following the redirection symbol corresponding to standard error:

 ./my_prog  1> output  2>&1
	

C Shell

If you are using the C Shell, use the greater than symbol (>) to specify the file to be used for standard output and the greater than symbol followed by an ampersand (>&) to specify the file to be used for standard error. For example, to send the standard output from the program "my_prog" to a file called "my_prog.out" and the error diagnostics to a file called "my_prog.err", enter the following command:

./my_prog  >my_prog.out  >&my_prog.err
	

Sometimes it is desirable to include the error diagnostics in the same file as the standard output. To write both the standard output and error diagnostics from "my_prog" to my_prog.out", enter:

 ./my_prog  >&my_prog.out
	

Background Processing

When you are busy, you may not want to wait until a command finishes processing before you enter your next command. You can begin a process in the background by including an ampersand (&) following the command; you can then continue working by entering new commands. For example, to compile 2cent.c in the background, enter:

cc  -o  2cent  2cent.c  &
	ls
	

Background Jobs and Logout -- the "nohup" command

In the prior section, you learned how to run a job in the background while you continued to execute other programs; however, background processing is typically terminated when you logout. To enable a program to continue running in the background even after logging off, precede the command you wish to run in the background with the command "nohup".

Bourne and Korn Shells

By default, the program output (standard output and standard error) will be placed in the file "nohup.out" in the working directory. If you like, you may specify files for redirecting the output (see also Redirecting Standard Error). For example, to submit a background job "my_prog" for execution, enable it to run after you logout, and to send the standard output to file "my_prog.out" and standard error to "my_prog.err", enter:

nohup  ./my_prog  1> my_prog.out  2> my_prog.err  &
	

C Shell

The C Shell includes its own internal "nohup" command. This internal command does not create a default "nohup.out" file for program output; you must redirect the output to files if you wish to have it preserved across sessions. For example, to submit a background job "my_prog" for execution, enable it to run after you logout, and to send the standard output and standard error to "my_prog.out", enter:

nohup  ./my_prog >&my_prog.out &
	

Changing Priority: nice

If you are working on a shared system or if you are running multiple cpu intensive jobs, you may wish to assign a lower priority to some jobs so that time critical jobs can be done as soon as possible and yet allow the low priority jobs to execute using the remaining cpu cycles. The "nice" command is used to indicate that you would like these jobs to be "nice" relative to the rest of the system, i.e., you would like them to be executed at a lower priority. For example, to run "my_prog" in the background at a lower priority, enter:

nice  ./my_prog &
	

Monitoring Process Status

You can list your current processes by entering the ps command. If you have logged off and logged back on, you can list the current status of your jobs by entering one of the following commands:

ps aux | grep $USER
	       or
	ps aux | more
	

The "aux" option is necessary if you have logged off and logged back on; otherwise only the processes started during the current session will be displayed.

Using the tail Command to Monitor Progress of a Background Job

tail  -f  ./my_prog.out
	

The "-f" option continually displays the contents of the file as new information is added and you can thus watch the output from a background job as it is generated. You can terminate the display of the program output by pressing <Ctrl-C> to cancel the tail command.

Terminating a Process

After you have listed the current processes, you have available the process id numbers. You can then use the kill command with the process id number of one of your jobs to specify the job you wish to cancel:

kill my_prog_process_number

The C and Korn Shells provide capabilities for changing the status of a process. Thus it is possible to place a job which is already executing in the foreground into the background or to bring a background job into the interactive environment. The shell commands "bg" and "fg" are used to move processes to the background and foreground. To move an active job into the background, first suspend it by pressing <Ctrl-Z> and then enter the command "bg". To bring a background job into the foreground, use the ps command to obtain its process control id number and then enter:

fg  process_id_number
	

Obtaining Execution Times

The "time" command can be used to write program execution time statistics (CPU and wall clock time) to standard error. The system "time" command returns the following values:

real

elapsed clock time in seconds

user

user process time in seconds

sys

CPU time in seconds

Bourne and Korn Shells

To execute "my_prog" in the background, to write standard output "myprog.out", and to write any error diagnostics and the timing statistics to "my_prog.err", enter:

time ./my_prog  1> my_prog.out  2> my_prog.err &
	

time ./my_prog  1> my_prog.out  2>&1 &
	

C Shell

The C Shell includes its own internal "time" command which generates different output from that of the system "time" command. If you are using the C Shell, you have a choice of which "time" command to use. The internal command will be used unless you specify the path (typically "/bin") for the system time command.

To execute "my_prog" in the background, to write standard output "myprog.out", and to write any error diagnostics and the timing statistics (from the system "time" command) to "my_prog.err", enter:

(/bin/time ./my_prog >my_prog.out) >&my_prog.err &
	

/bin/time ./my_prog >&my_prog.out &
	

Using the Profiler to Determine Time Spent in SubProgram Components

The -O (optimize) compiler option requests that the compiler optimize the executable code for best performance. If you are writing a program which will be executed many times, you can sometimes obtain even greater performance by rewriting portions of the code. The profiler can be used to identify those portions of your code which require the most processing time and thus suggest sections of code which might hold the greatest promise for revision.

The profiler is easy to use: first include the -p option when you compile your code, execute the program, and then run the program "prof". The following example illustrates how the profiler could be used with the sample FORTRAN program included in the xmp subdirectory:

 xlf -p -ostatsall statmain.f means.f std.f
	 ./statsall
	 prof > prof.out
	

If a routine is called multiple times within your program, use "-pg" instead of "-p" in the above example and "gprof" instead "prof". The g profiler provides extended diagnostics which will help you identify which calls to a subprogram were the ones which were most time consuming. Both profilers create large intermediate output files (may exceed your disk quota). After you are finished your analysis, you should delete the file mon.out (gmon.out if you are using the g profiler). Once you are finished with the profile analysis, you should also recompile your code without the "-p" (or "-pg") option.

UNIX Program Development Tools

Some of the tools which make UNIX particularly attractive to program developers include:

awk

search and process patterns within files

lint

syntax checker for C code portability across systems

make

maintains programs in most current form.

sccs

Source Code Control System which provides management and tracking of code or text document development.

cpp

performs file inclusion and macro substitution in C language source files.

m4

processes files with expansion of macro definitions

curses

provides control of cursor positioning and window usage.

sed

text stream (non-interactive) editor

Using the make Utility

The make command is used in program development to simplify the process of compilation when a program consists of several subprogram components. Rather than recompiling every routine when the code is changed, the make program rebuilds only those object code segments which would be affected by the changes. This capability becomes particularly useful when the code contains several include files.

The following exercise illustrates use of make with a FORTRAN program consisting of a main program, two subprograms, and an include file used in the main program and one of the subprograms.

1. Examine the contents of "makefile" and read the comments which describe each of the commands contained in the file.

   Note: The indentations at the beginning of lines in a makefile must be created by 'tab' characters and not by spaces.

2. Generate and execute a program from multiple FORTRAN source files:
   • make
   • ./statsall
3. Edit (vi) "included" and change the value of MVARS to 5. Repeat step 1 above and note:
   • subroutine var was not recompiled as it is not dependent upon 'included'.
   • The output now consists of 5 lines instead of just one.

A second example makefile is provided in the file "makefile.f". This makefile performs the same functions, but includes additional statements to define make variables to provide a more portable file for use with other operating systems. Simply by changing the values of the variables "compile", "link", and "librarian" to the appropriate commands and command options, this makefile can be used with Microsoft FORTRAN as well as with FORTRAN compilers on other UNIX systems. To use with other UNIX systems, simply change "xlf" to "f77".

The make command uses the file "makefile", "MAKEFILE", or MakeFile" as a source file of commands used to build a program. The "-f" option of the make command is used to specify the name of a file containing "make" commands which is to be used to build a program. For example, to compile and then execute the program "statsall" created by using the commands in the file "makefile.f", enter:

 make  -f  makefile.c
	 ./statsall
	

A third example makefile is provided in the "C" subdirectory that you copied from aixstu00. Change into this directory and enter the following command to build and execute a new version of the "statsall" executable from c source code:

 make  -f  makefile.c
	 ./statsall
	

Using the Archive Librarian

The ar command is used to create and manage archive libraries. These libraries may contain source files or compiled object code.

The command flags 'vru' can be used with the ar command to create a new archive library or to update and replace members in an existing library. To create the archive library 'libstats' from the object code files means.o and std.o, you could use the following command:

ar  vru  libstats.a  means.o  std.o
	

xlf  -o  statsall  statmain.f  -L.  -lstats
	

The example file 'makefile.f' uses the above command to maintain the object code for two subroutines which it requires.

A wide variety of FORTRAN, C, and other programming language source files are available by anonymous ftp on the Internet. If you use some of these routines frequently, you may find it advantageous to compile and store the compiled code in an archive library.

Note: Archive libraries, including source code only archive libraries, are not portable across UNIX systems. If you are using more than one UNIX system, you will typically will need to recreate archive libraries on each of the systems where you would like to use them.

Some users of VTAIX have set permissions on their library archives to allow others to use these libraries. The ar command can also be used to list and extract the members of archive libraries. For example, to list the subroutines present in libstats.a, enter:

ar  -vt  libstats.a
	

To extract the object code for 'means.o' from 'libstats.a', enter:

ar  -vx  libstats.a  means.o
	

To delete a library member, use the '-d' option flag of the ar command.

Mixed Language Programming: Timing Portions of FORTRAN Code Using a C Language Function

The examples illustrated at the end of this section include a shell script which can be executed under the AIX operating system to compile the C function cclock, compile a Fortran program, link the two sets of object code, and then run the resultant module.

The example files illustrated in these figures are found in the /xmp/mixed_lang subdirectory copied to your disk with the sample files. The option -lc is used with the xlf command to include the C language libraries required by the timing routine when the two programs are linked. As seen in this example, it is easy to combine object code from different compilers on VTAIX. You must be careful about passing arguments between different languages, but if this is done correctly, all you must then do is to compile the code using the corresponding compiler with the -c (compile only) option and then link (xlf in the example) with the required language libraries.

On other systems, mixed language programming may not be this easy. Additional code within the calling and called programs may be required to properly define the languages being used or a specific nomenclature may be require for functions which are to be called by another language; see your system's documentation for details.

Shell Program "test.sh" -- This shell script is used to to compile a FORTRAN program which invokes a C function to obtain timing statistics.

#!/bin/sh
	# Shell script to Compile a FORTRAN Program which
	# Invokes a C Function xlc -c cclock.c
	xlf -o myprog progname.f cclock.o -lc
	/bin/time ./myprog
	

	C Function "cclock.c"
	

	/*  C function to compute elapsed CPU time using the UNIX clock function */
	
	/* This C function returns CPU utilization in milliseconds   */
	/*      It is, however, only accurate to about 1/100 second  */
	
	/* To access from a FORTRAN program, use:  CALL cclock(isec) */
	/*    where isec is INTEGER*4                                */
	
	#include <time.h>
	void cclock(long int *cpu_sec)
	{
	  *cpu_sec = clock();
	}
	

	C ---- Example FORTRAN Program "progname.f"
	C
	C      This FORTRAN program calls the C language function to obtain
	C      timing statistics.
	C
	       INTEGER*4  TIME
	       CALL cclock(time)
	       WRITE(6,*) 'CPU Time Utilization is:  ',time/1E+6,'  Seconds'
	       do 1000 i=1, 1000000
	          x = i
	          y = x * x
	 1000  Continue
	       CALL cclock(time)
	       WRITE(6,*) 'CPU Time Utilization is:  ',time/1E+6,'  Seconds'
	       do 2000 i=1,2400000
	          x = i
	          y = x * x
	 2000  Continue
	       CALL cclock(time)
	       WRITE(6,*) 'CPU Time Utilization is:  ',time/1E+6,'  Seconds'
	       CALL cclock(time)
	       WRITE(6,*) 'CPU Time Utilization is:  ',time/1E+6,'  Seconds'
	       END