Compiler 2016: Testing Environement Specification - midterm

How a submission is judged in this phase

Boot a virtual machine in vbox of Ubuntu 14.04 with Oracle Java 1.8.0_77
git clone one's compiler2016
(IMPORTANT)git checkout midterm, the version with the 'midterm' tag is literally the most important one. Do push tags!
To judge this submission (e.g. abc/compiler2016/)
1. Kill all irrelevant processes
2. Setup a new bash process
3. Reset environment variables
4. cd abc/compiler2016/
5. make clean # delete the bin dir
6. make all # compile your code, please put your executables in a bin directory. A makefile is required!
7. source midtermvars.sh # setup environment variables for mid-term testing. Define $CCHK in this script.
8. cd bin
9. For each test case, copy the test case to abc/compiler2016/bin/data.mx
  1. $CCHK < data.mx
  2. Check $? (the exit status); 0 for OK, 1 for error
  3. Don't output too much debug information to standard output in your submission. Please only return 0 or 1.

Notes

You need a Makefile for this mid-term testing. If you don't know how to write a Makefile, read http://mrbook.org/tutorials/make/

Note that you also need a Makefile even if you want to compile Java using ant or maven.

For each test case, your compiler is required to exit normally. Write operations to the file system are prohibited.

You can setup environment variables as follows, according to your Makefile:

export CCHK=java -cp .:classes/ cc.midterm.Main

or

export CCHK=java -jar midterm.jar

In Java, use System.exit(...) to specify the exit status.

Testing Script

#!/bin/bash

echo 'please modify the $names variable to be your own id'
names="$(cat students_list.txt)" #modify to be your own id
echo $names
timeo=10s

function _SC() #safe call
{
  echo "[RUNNING] : $*"
  eval $*
  if [ $? -ne 0 ]; then
    echo "[ERROR] : command $* failed, please check, exiting..."
    exit
  fi
}

outdir=$(pwd)

mkdir -p students

if [ ! -d compiler2016-testcases ]; then
  _SC git clone https://bitbucket.org/acmcompiler/compiler2016-testcases.git
else
  _SC cd compiler2016-testcases
  _SC git pull
  _SC cd ..
fi
cedir=$(pwd)/compiler2016-testcases/compile_error
normaldir=$(pwd)/compiler2016-testcases/passed
mkdir -p results
resdir=$(pwd)/results

for name in ${names}; do
  echo now testing ${name}...
  _SC cd students
  if [ ! -d $name ]; then
    _SC mkdir $name "&&" cd $name
    _SC git clone "https://bitbucket.org/${name}/compiler2016.git"
    _SC cd ..
  fi
  _SC cd $name/compiler2016
  _SC git checkout master
  _SC git pull
  _SC git checkout midterm #pay attention to your tag!
  _SC make clean
  _SC make all
  if [ ! -e ./midtermvars.sh ]; then
    echo "[ERROR] : can't find midtermvars.sh, exiting..."
    exit
  fi

  unset -v CCHK
  set -x
  source ./midtermvars.sh #can't ensure SC here
  set +x
  echo CCHK=$CCHK

  rm $resdir/$name.LOG
  _SC cd bin
  for tcase in $(ls $cedir/*.mx); do
    _SC cp $tcase data.mx
    echo "now $tcase" >> $resdir/$name.LOG
    (timeout $timeo $CCHK <data.mx) 1>>$resdir/$name.LOG 2>&1
    if [ $? -ne 1 ]; then
      echo "[WRONG] : wrong!!"
      echo "No" >> $resdir/$name.LOG
      sleep 1
    else
      echo "Yes" >> $resdir/$name.LOG
    fi
  done

  for tcase in $(ls $normaldir/*.mx); do
    _SC cp $tcase data.mx
    echo "now $tcase" >> $resdir/$name.LOG
    (timeout $timeo $CCHK <data.mx) 1>>$resdir/$name.LOG 2>&1
    if [ $? -ne 0 ]; then
      echo "[WRONG] : wrong!!"
      echo "No" >> $resdir/$name.LOG
      sleep 1
    else
      echo "Yes" >> $resdir/$name.LOG
    fi
  done

  echo "LOG saved to $resdir/$name.LOG"

  _SC cd ..
  _SC git checkout master
  cd $outdir
done