Compiler 2014: Testing environment specification

Testing Process

Boot an up-to-date Ubuntu Desktop 12.04 system in VirtualBox
Install spim from https://bitbucket.org/xjia/statspim
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
java version "1.7.0_51"
jflex 1.5.1
git clone your repository
(IMPORTANT)git checkout finalterm, we regard the 'finalterm' tag as your submitted version, so don't forget to push the tags.
For each repository (e.g. xjia/compiler2014/)
1. Kill all irrelevant processes
2. Setup a new bash process
3. Reset environment variables
4. cd xjia/compiler2013/
5. make clean # delete the bin dir
6. make all # compile your code, plese put your executables in a bin directory
7. source finaltermvars.sh # setup environment variables for final-term testing, you might as well copy some auxiliary file to the bin directory
8. cd bin
9. For each test case, copy the test case to xjia/compiler2013/bin/data.c
  1. $CCHK data.c 1>assem.s
  2. Check $? (the exit status); 0 for OK, 1 for error
  3. spim -stat -file assem.s
  4. We will redirect your standard output to an file, so don't output any debug information to standard output in testing.

How the assemble is runed

./spim -stat -file assem.s

In the last line, how many executed instructions are shown.

Limits

For each test case, your compiler is required to exit within 5 seconds. Write operations to the file system are prohibited. But you are allowed to write to the standard output (stdout or System.out) and the standard error (stderr or System.err) stream.

Notes

You can setup environment variables like this:

export CCHK=java -cp .:classes/ cc.midterm.Main

or

export CCHK=java -jar midterm.jar

or

export CCHK=./c_compiler

In Java, use System.exit(...) to specify the exit status. In C, use exit(...) to specify the exit status.

You need a Makefile for this mid-term testing. If you don't know how to write a Makefile, read http://mrbook.org/tutorials/make/

Note that you also need a Makefile even if you want to compile Java using ant or maven.

Standard-library Functions

scanf
printf
malloc

About git tags

tag the current commit

 git tag -a tagname -m "message"

delete the tag locally

 git tag -d tagname

to see about a tag, use 'git show tagname' or see it in bitbucket web's commit board.

tags need to be pushed to the server use 'push --tags'

 git push --tags

or

 git push origin master --tags

remember to use 'git fetch' to pull the update of tags

 git fetch --tags

use 'git checkout' to go back to some commit(tag), after that, use 'git checkout master' to come back~

 git checkout tagname

A toy testing script

put it into an empty directory
change the names="acmcompiler" line to be your own bitbucket id
run it

#!/bin/bash

echo 'please modify the $names variable to be your own id'
echo 'please make sure build-answers.sh is run in the normaldata directory'
names="acmcompiler" #modify to be your own id
#names=$(cat names)
echo $names
sleep 2
timeo=10s


function _SC() #safe call
{
  echo "[RUNNING] : $*"
  eval $*
  if [ $? -ne 0 ]; then
    echo "[ERROR] : command $* failed, please check, exiting..."
    exit
  fi
}

outdir=$(pwd)

mkdir -p students

if [ ! -d compiler2014-testcases ]; then
  _SC git clone git@bitbucket.org:acmcompiler/compiler2014-testcases.git
else
  _SC cd compiler2014-testcases
  _SC git pull
  _SC cd ..
fi

normaldir=$(pwd)/compiler2014-testcases/Normal

for name in ${names[@]}; do
  cd $outdir
  score=0
  echo now testing ${name}...
  _SC cd students
  if [ ! -d $name ]; then
    _SC mkdir $name "&&" cd $name
    _SC git clone "git@bitbucket.org:${name}/compiler2014.git"
    _SC cd ..
  fi
  _SC cd $name/compiler2014
  _SC git checkout -f master
  _SC git pull 
  _SC git fetch --tags
  _SC git checkout -f finalterm #pay attention to your tag! someone don't have the midterm tag
  _SC make clean #some one don't have make clean
  _SC make
  _SC cat ./finaltermvars.sh

  unset -v CCHK
  set -x
  source ./finaltermvars.sh #can't ensure SC here 
  set +x
  echo CCHK=$CCHK

  wrong=0
  _SC cd bin 
  for filec in $(ls $normaldir/*.c); do
	filea=${filec%.c}.ans
	cp $filec data.c
	timeout $timeo $CCHK data.c 1>assem.s
	spim -stat_file spimstat -file assem.s 1>spimout
	diff spimout $filea
	num=-1
	num=$(cat spimstat | awk ' { print $1 } ')
	echo NUMBER : $num
	sleep 1
  done
  _SC cd ..
  _SC git checkout -f master
  cd $outdir
done