PPCA 2016: Testing Environement Specification MIPS

How a submission is judged in this phase

Boot a virtual machine in vbox of Ubuntu 16.04 LTS with Oracle Java 1.8.0_91
git clone one's mips-simulator
(IMPORTANT)git checkout final, the version with the 'final' tag is literally the most important one. Do push tags!
To judge this submission (e.g. abc/mips-simulator/)
1. Kill all irrelevant processes
2. Setup a new bash process
3. Reset environment variables
4. cd abc/mips-simulator/
5. make clean # delete the bin dir
6. make all # compile your code, please put your executables in a bin directory. A makefile is required!
7. source setvars.sh # setup environment variables for mid-term testing. Define $MSCK in this script.
8. cd bin
9. For each test case data.s
  1. $MSCK data.s < data.in > data.out
  2. compare data.out with data.ans
  3. Don't output debug information to standard output in your submission. Please print your assembly only.

Notes

You need a Makefile for this testing. If you don't know how to write a Makefile, read http://mrbook.org/tutorials/make/

Note that you also need a Makefile even if you want to compile Java using ant or maven.

For each test case, your simulator is required to exit normally. Write operations to the file system are prohibited.

You can setup environment variables as follows, according to your Makefile, like:

export MSCK="java -cp .:classes/ cc.final.Main"

or

export MSCK="java -jar final.jar"

or

export MSCK=./final

Testing Script

#!/bin/bash

echo 'please modify the $names variable to be your own id'
names=$(cat list.txt) #modify to be your own id
echo $names
sleep 2
timeo=180s

function _SC() #safe call
{
    echo "[RUNNING] : $*"
    eval $*
    if [ $? -ne 0 ]; then
        echo "[ERROR] : command $* failed, please check, exiting..."
        exit
    fi
}

outdir=$(pwd)

mkdir -p students

if [ ! -d mips-simulator-testcases ]; then
    _SC git clone git@github.com:BreakVoid/mips-simulator-testcases.git
else
    _SC cd mips-simulator-testcases
    _SC git pull
    _SC cd ..
fi

normaldir=$(pwd)/mips-simulator-testcases/testsuit-1
resdir=$(pwd)/results
mkdir -p $resdir
statistics=$resdir/statistics
#rm -rf $statistics

for name in ${names[@]}; do
    cd $outdir
    diff_result_dir=$resdir/diff_result_$name
    rm -rf $diff_result_dir
    mkdir -p $diff_result_dir
    log_file=$resdir/${name}.LOG
    rm -rf $log_file
    score=0
    full_score=0
    echo now testing ${name}...
    _SC cd students
    if [ ! -d $name ]; then
        _SC mkdir $name "&&" cd $name
        _SC git clone "git@bitbucket.org:${name}/mips-simulator.git"
        _SC cd ..
    fi
    _SC cd $name/mips-simulator
    _SC git checkout -f master
    _SC git pull 
    _SC git fetch --tags
    _SC git checkout -f final #pay attention to your tag! someone don't have the midterm tag
    _SC make clean #some one don't have make clean
    _SC make
    _SC cat ./setvars.sh
 
    unset -v MSCK
    set -x
    source ./setvars.sh #can't ensure SC here 
    set +x
    echo MSCK=$MSCK
 
    _SC cd bin 
    for filec in $(ls $normaldir/*.s); do
        filein=${filec%.s}.in
        fileout=${filec%.s}.ans
        _SC cp $filec data.s
        if [ -f $filein ]; then 
            timeout $timeo $MSCK data.s 1>msckout 2>/dev/null
        else
            timeout $timeo $MSCK data.s <$filein 1>msckout 2>/dev/null
        fi
        pure_file_name=$(basename $filec)
        pure_file_name=${pure_file_name%.s}
        diff msckout $fileout >$diff_result_dir/$pure_file_name
        full_score=$((full_score+1))
        file_limit=${filec%.mx}.limit
        if [ ! -s $diff_result_dir/$pure_file_name ]; then
            score=$((score+1))
        else
            echo ${filec%.s} : FAILED >>$log_file
            echo FAILED
        fi
        sleep 1
    done
    echo count: $score/$full_score >>$log_file
    echo count: $score/$full_score
    echo $name $score/$full_score >>$statistics
    _SC cd ..
    _SC git checkout -f master
    cd $outdir
done