SAT-Race 2008

SAT-Race
2008

Overview
Rules
Important Dates
Organizers
Qualification
Solvers
Results
Result Analysis
Downloads

Results

This page presents the results of SAT-Race 2008. The winners in the three tracks (Main, Special 1, Special 2) are:

Track	First Place	Second Place	Third Place
Main Track (CNF sequential)	MiniSat 2.1	pMiniSat	Barcelogic
Special Track 1 (CNF parallel)	ManySat	pMiniSat	MiraXT
Special Track 2 (AIG sequential)	MiniSat++ 1.0	picoaigersat	MiniCirc

The prize for the best student solver goes to pMiniSat.

Detailed results on each track are given below.

Main Track (CNF sequential)

The table below shows the number of instances solved by each SAT solver (out of 100 instances in total; also split up into successfully processed SAT and UNSAT instances); the rank according to number of solved instances; number of speed-points assigned to solver (see the Rules page for a description of speed-points); the rank according to speed-points; the total score; and the final ranking based on the total score. The time-out for each solver and each instance was 900 seconds.

	MiniSat 2.1	pMiniSat*	Barcelogic	Rsat	MXC	preSAT	CMUSAT	kw	picosat	Eureka
#solved:	81	79	77	76	74	72	72	70	70	69
#solved SAT/UNSAT:	43/38	39/40	35/42	36/40	40/34	38/34	40/32	32/38	40/30	29/40
rank by solved:	1	2	3	4	5	6	6	8	8	10
speed points:	7.26	5.61	5.10	5.04	4.79	4.73	4.72	5.19	5.16	5.44
rank by speed:	1	2	6	7	8	9	10	4	5	3
total score:	88.26	84.61	82.10	81.04	78.79	76.73	76.72	75.19	75.16	74.44
rank:	1	2	3	4	5	6	7	8	9	10

	ManySat*	LocalMinisat	MiraXT*	SATzilla	eSAT	Spear	Tinisat	clasp	SAT4J
#solved:	68	64	63	60	60	60	56	54	49
#solved SAT/UNSAT:	30/38	32/32	37/26	30/30	29/31	30/30	28/28	23/31	26/23
rank by solved:	11	12	13	14	14	14	17	18	19
speed points:	3.99	3.68	4.26	4.18	3.81	3.32	3.08	3.19	2.43
rank by speed:	13	15	11	12	14	16	18	17	19
total score:	71.99	67.68	67.26	64.18	63.81	63.32	59.08	57.19	51.43
rank:	11	12	13	14	15	16	17	18	19

^*Note: pMiniSat, ManySat and MiraXT are parallel solvers, which were run in sequential mode.

Detailed results including runtimes for all solvers and instances can be viewed in HTML or downloaded in MS-Excel format.

We have also run the best three solvers of 2007's SAT-Competition (Industrial Track, SAT+UNSAT Category) on the 100 instances of SAT-Race 2008. The results are available in both HTML and MS-Excel format.

Special Track 1 (CNF parallel)

In the parallel track, each solver could use four cores (two Dual-Core Intel Xeon CPUs). As runtimes are highly deviating for parallel SAT solvers, each solver was run three times on each instance. An instances was considered solved, if it could be solved in at least one of the three runs of a solver.

The table below gives the number of solved instances (also separated by SAT and UNSAT instances), as well as the number of instances that could be solved in all three runs, and the number of instances that could be solved in less than 900 seconds of CPU time.

	ManySat	pMiniSat	MiraXT
#solved:	90	85	73
#solved SAT/UNSAT:	45/45	44/41	43/30
#solved in all runs:	79	84	66
#solved in ²900s CPU time:	69	72	53
rank:	1	2	3

Detailed results including runtimes for all solvers and instances can be viewed in HTML or downloaded in MS-Excel format. The runtimes in these tables are median wall-clock times avaraged over all successful runs. Additionally, for each instance and solver, the number of successful runs (between 0 and 3) is given.

Special Track 2 (AIG sequential)

The table below summarizes the results of the AIG track. The reported numbers have the same interpretation as in the Main Track.

	MiniSat++	pico-aigersat	MiniCirc	CMUSAT-AIG	kw_aiger	NFLSAT
#solved:	74	69	70	70	62	60
#solved SAT/UNSAT:	31/43	33/36	28/42	33/37	26/36	30/30
rank by solved:	1	4	2	2	5	6
speed points:	12.98	13.80	12.29	11.85	9.18	9.34
rank by speed:	2	1	3	4	6	5
total score:	86.98	82.80	92.29	81.85	71.18	69.34
rank:	1	2	3	4	5	6