| Testing Technique | Title | Language | Actions |
|---|---|---|---|
| Model based testing | A Modular Approach to Model-based Testing of Concurrent Programs | Java | |
| Structural testing | An Empirical Evaluation of the Cost and Effectiveness of Structural Testing Criteria for Concurrent Programs | C, MPI | |
| Model based testing | BALLERINA: automatic generation and clustering of efficient random unit tests for multithreaded code | Java | |
| Structural testing | CARISMA: a context-sensitive approach to race-condition sample-instance selection for multithreaded applications. | Java | |
| Fault-based testing | Comparative assessment of testing and model checking using program mutation. | Java | |
| Fault-based testing | Efficient mutation testing of multithreaded code | Java | |
| Formal method-based testing | Generating effective tests for concurrent programs via AI automated planning techniques | Java | |
| Structural testing | Generating unit tests for concurrent classes | Java | |
| Fault-based testing | How Good is Static Analysis at Finding Concurrency Bugs? | Java | |
| Structural testing | Location pairs: a test coverage metric for shared-memory concurrent programs | Java | |
| Structural testing | The Impact of Concurrent Coverage Metrics on Testing Effectiveness | Java | |
| Structural testing | A Lightweight and Portable Approach to Making Concurrent Failures Reproducible | Java | |
| structural testing | An approach to testing commercial embedded systems | C | |
| No Information | An Automation-Assisted Empirical Study on Lock Usage for Concurrent Programs | Java | |
| Fault-based testing | An empirical study of data race detector tools | Java | |
| Structural testing | Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation | Java | |
| Fault-based testing | ASP: Abstraction Subspace Partitioning for Detection of Atomicity Violations with an Empirical Study | Java | |
| Structural testing | Assertion Guided Symbolic Execution of Multithreaded Programs | C, C++ | |
| Fault-based testing | Automated Classification of Data Races Under Both Strong and Weak Memory Models | C, C++ | |
| No Information | Clash of the Titans: Tools and techniques for hunting bugs in concurrent programs | java, C# | |
| Formal method-based testing | Concurrency Testing Using Schedule Bounding: an Empirical Study | C, C++, Java | |
| Model based testing | ConLock: A Constraint-based Approach to Dynamic Checking on Deadlocks in Multithreaded Programs | C, C++, Java | |
| Fault-based testing | Detecting Event Anomalies in Event-based Systems | Java | |
| Model based testing | Effective random testing of concurrent programs | Java | |
| Structural testing | Empirical evaluation of a new composite approach to the coverage criteria and reachability testing of concurrent programs | C, MPI | |
| Structural testing | Incremental integration testing of concurrent programs | CCS, CSP, Ada | |
| Fault-based testing | Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. | C, C++ | |
| Fault-based testing | RACEZ : A Lightweight and Non-Invasive Race Detection Tool for Production Applications | C, C++ | |
| Structural testing | RECONTEST: Effective Regression Testing of Concurrent Programs | Java | |
| Fault-based testing | Runtime Prevention of Concurrency Related Type-state Violations in Multithreaded Applications | C, C++, Pthreads | |
| Structural testing | Saturation-based testing of concurrent programs | Java | |
| Fault-based testing | Selective mutation testing for concurrent code | Java | |
| Structural testing | Testing Concurrent Programs on Relaxed Memory Models | C | |
| Structural testing | Toward a Methodology to Expose Partially Fixed Concurrency Bugs in Modified Multithreaded Programs | Pthread | |
| Model based testing | Incremental testing of concurrent programs using value schedules | Java | |
| Model based testing | Efficient Deterministic Multithreading Without Global Barriers | C, C++, Pthreads | |
| Structural testing | Efficient Execution Path Exploration for Detecting Races in Concurrent Programs | Java | |
| Structural testing | Pegasus: Automatic Barrier Inference for Stable Multithreaded Systems | C, C++ | |
| Fault-based testing | Race directed random testing of concurrent programs | Java | |
| Formal method-based testing | SATCheck: SAT-Directed Stateless Model Checking for SC and TSO | C | |
| Fault-based testing | TRADE: Precise Dynamic Race Detection for Scalable Transactional Memory Systems | C, C++ | |
| Structural testing | A combinatorial testing strategy for concurrent programs | Java | |
| Structural testing | A meta heuristic for effectively detecting concurrency errors | Java | |
| Structural testing | A model-free and state-cover testing scheme for semaphore-based and shared-memory concurrent programs | Java | |
| Structural testing | A platform for search-based testing of concurrent software | Java | |
| Structural testing | A race-detection and flipping algorithm for automated testing of multi-threaded programs | Java | |
| Structural testing | A randomized scheduler with probabilistic guarantees of finding bugs | C, C++ | |
| Fault-based testing | Accurate and Efficient Runtime Detection of Atomicity Errors in Concurrent Programs | Java | |
| Structural testing | Advances in noise-based testing of concurrent software | C, Java | |
| Structural testing | An Efficient and Flexible Deterministic Framework for Multithreaded Programs | Pthread | |
| Fault-based testing | An Evaluation of Data Race Detectors Using Bug Repositories | Java | |
| Fault-based testing | ASN: A Dynamic Barrier-Based Approach to Confirmation of Deadlocks from Warnings for Large-Scale Multithreaded Programs | C, C++, Java | |
| Structural testing | Automatic testing environment for multi-core embedded software - ATEMES | C, C++ | |
| Structural testing | Bita: Coverage-guided, automatic testing of actor programs | Scala | |
| Fault-based testing | CLAP: Recording Local Executions to Reproduce Concurrency Failures | C, C++ | |
| Formal method-based testing | Composable Specifications for Structured Shared-Memory Communication | Java | |
| No Information | Con2Colic Testing | Java | |
| Formal method-based testing | ConcBugAssist: Constraint Solving for Diagnosis and Repair of Concurrency Bugs | C | |
| Model based testing | Concurrency Debugging with Differential Schedule Projections | C, C++, Java | |
| Formal method-based testing | Concurrent software verification with states, events, and deadlocks | C | |
| Fault-based testing | ConMem: Detecting severe concurrency bugs through an effect-oriented approach | C, C++ | |
| Fault-based testing | ConSeq: Detecting Concurrency Bugs through Sequential Errors | C, C++ | |
| No Information | Coverage guided systematic concurrency testing | C, C++, Pthreads | |
| Structural testing | Coverage metrics for saturation-based and search-based testing of concurrent software | Java | |
| Fault-based testing | CTrigger: Exposing atomicity violation bugs from their hiding places | C, C++ | |
| Model based testing | Distributed reachability testing of concurrent programs | Java | |
| Fault-based testing | Effective and Precise Dynamic Detection of Hidden Races for Java Programs | Lotus, Java | |
| Formal method-based testing | Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool | C, C++, Pthreads | |
| Structural testing | Enforcer - Efficient failure injection. | Java | |
| Model based testing | Exploiting parallelism in deterministic shared memory multiprocessing | Java | |
| Fault-based testing | Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing | C, C++ | |
| Formal method-based testing | Iterative context bounding for systematic testing of multithreaded programs | C, C++, C# | |
| Structural testing | Kendo: efficient deterministic multithreading in software | C, C++ | |
| Fault-based testing | Kivati: Fast detection and prevention of atomicity violations | Pthread, C | |
| Structural testing | Maple: a coverage-driven testing tool for multithreaded programs | C, C++ | |
| Structural testing | Maximizing synchronization coverage via controlling thread schedule | Pthread | |
| Structural testing | Multicore Acceleration of Priority-Based Schedulers for Concurrency Bug Detection | C++ | |
| Model based testing | MultiOtter : Multiprocess Symbolic Execution | C | |
| Fault-based testing | Multithreaded Test Synthesis for Deadlock Detection | Java | |
| Structural testing | On a Technique for Transparently Empowering Classical Compiler Optimizations on Multithreaded Code | C, C++ | |
| Fault-based testing | Race Detection for Android Applications | Java | |
| No Information | Reachability testing of concurrent programs. | C++ | |
| Fault-based testing | Runtime analysis of atomicity for multithreaded programs. | Java | |
| Fault-based testing | SOS: Saving Time in Dynamic Race Detection with Stationary Analysis | Java | |
| Functional testing | Test-Data Generation for Testing Parallel Real-Time Systems | C | |
| Formal method-based testing | TM UNIT : A Transactional Memory Unit Testing and Workload Generation Tool | Pthread, C | |
| Formal method-based testing | Variable and thread bounding for systematic testing of multithreaded programs | Java, C# | |
| Model based testing | Applications of Model Reuse When Using Estimation of Distribution Algorithms to Test Concurrent Software | Java | |
| Structural testing | Value-deterministic Search-based Replay for Android Multithreaded Applications | Java | |
| Functional testing, Formal method-based testing | On the Effectiveness of Contracts as Test Oracles in the Detection and Diagnosis of Functional Faults in Concurrent Object-Orien | Java | |
| Formal method-based testing | Types for safe locking: Static race detection for Java | Java | |
| Fault-based testing | AVIO: Detecting atomicity violations via access interleaving invariants | C, C++ | |
| Structural testing | Detection of asynchronous message passing errors using static analysis | Erlang | |
| Structural testing | DTAM: Dynamic Taint Analysis of Multi-threaded Programs for Relevancy Malay Ganai. | C, C++ | |
| Structural testing | Efficient concurrency-bug detection across inputs | C, C++ | |
| Fault-based testing | Efficient data race detection for distributed memory parallel programs | C, UPC | |
| Structural testing | InstantCheck: Checking the determinism of parallel programs using on-the-fly incremental hashing | C++ | |
| Fault-based testing | MuTMuT: Efficient Exploration for Mutation Testing of Multithreaded Code. | Java | |
| Structural testing | SAM: Self-adaptive dynamic analysis for multithreaded programs | Java | |
| Fault-based testing | Sound Predictive Race Detection in Polynomial Time | Java | |
| Structural testing | Synchronization identification through on-the-fly test | Pthread | |
| Fault-based testing | Synthesizing Racy Tests | Java | |
| Fault-based testing | Synthesizing Tests for Detecting Atomicity Violations | Java | |
| Formal method-based testing | TransDPOR: A novel dynamic partial-order reduction technique for testing actor programs | ActorFoundry | |
| Model based testing | Unfolding based automated testing of multithreaded programs | Java | |
| Formal method-based testing | Using SPIN for automated debugging of infinite executions of Java programs | Java | |
| Formal method-based testing | Fair stateless model checking | C++ | |
| Structural testing | Verification and Coverage of Message Passing Multicore Applications | MCAPI | |
| Formal method-based testing | Preemption sealing for efficient concurrency testing | .Net | |
| Structural testing | Concurrent program verification with lazy sequentialization and interval analysis | C | |
| Structural testing | SPDebugger: A Fine-Grained Deterministic Debugger for Concurrency Code | Java | |
| Functional testing | Effectively manifesting concurrency bugs in android apps | Android | |
| Structural testing | DFTracker: detecting double-fetch bugs by multi-taint parallel tracking | C, C++ | |
| Structural testing | RaceTest: harmful data race detection based on testing technology in WS-BPEL | Java, C++ | |
| Structural testing | Testing Multi-Threaded Applications Using Answer Set Programming | Java | |
| Structural testing | Dynamic testing for deadlocks via constraints | Java, C, C++ | |
| Functional testing | Concurrency Bug Avoiding Based on Optimized Software Transactional Memory | C, C++ | |
| Structural testing | Thread-local concurrency: a technique to handle data race detection at programming model abstraction | MPI | |
| Fault-based testing | A Splitting Strategy for Testing Concurrent Programs | Java | |
| Structural testing | Using Delta Debugging to Minimize Stress Tests for Concurrent Data Structures | Java | |
| Structural testing | Efficient scalable thread-safety-violation detection: finding thousands of concurrency bugs during testing | .Net, C# | |
| Structural testing | Efficient verification of concurrent systems using synchronisation analysis and SAT/SMT solving | No Information | |
| Structural testing | AVPredictor: Comprehensive prediction and detection of atomicity violations | C, C++ | |
| Structural testing | Parallel bug-finding in concurrent programs via reduced interleaving instances | C |
A Modular Approach to Model-based Testing of Concurrent Programs
No Information
Distributed systems
Java
Model based testing
Thread interaction models
system
reachability testing
LTS model
partially-ordered sequences, totally-ordered sequences, modular sequences
effectiveness
Case study
Dynamic
Message passing
deterministic execution
abstract LTS model
No Information
incremental reachability analysis and new ATL reduction algoritm
bugs inserted by the researcher
No Information
No Information
No Information
An Empirical Evaluation of the Cost and Effectiveness of Structural Testing Criteria for Concurrent Programs
Linux
HPC
C, MPI
Structural testing
coverage criteria
unit
Execution trace
concurrent code in MPI
number of test cases to cover a criterion, probability to satisfy a testing criterion using a test set adequate to another testing criterion, percentage of defects detected
effectiveness, cost, strength
Experiment
Dynamic
Message passing
controlled execution
Parallel Control Flow Graph (PCFG)
check-point insertion
check point insertion
mutation based errors
ValiMPI
Academic
Linux
BALLERINA: automatic generation and clustering of efficient random unit tests for multithreaded code
No Information
HPC
Java
Model based testing
Random generation
unit
Linearizability
multithread code class, specification of the class
number of transitions
effectiveness
Experiment
Dynamic
Message passing
stateless exploration
No Information
No Information
clustering
bug-triggering interleavings, deadlock, exception, starvation
Ballerina
Academic
No Information
CARISMA: a context-sensitive approach to race-condition sample-instance selection for multithreaded applications.
Unix
HPC
Java
Structural testing
context-sensitive sampling
unit
happens-before relation, locksets
multithreaded program
performance: detection rate, effectiveness: detection rate, time overhead: slowdown of the strategy and space , overhead: memory consumption
effectiveness
Case study
Dynamic
Shared memory
monitoring
No Information
RoadRunner framework
probabilistic sampling
data race
Carisma
Academic
Unix
Comparative assessment of testing and model checking using program mutation.
Linux
HPC
Java
Fault-based testing
Mutation operators
unit
synchronization events, thread schedules
source code in Java
number of faults detected
effectiveness
Experiment
Dynamic
Shared memory
deterministic replay
No Information
manual instrumentation
parallel randomized state-space search
assertion violation, deadlock
ExMan, Contest, JPF
Academic
Linux
Efficient mutation testing of multithreaded code
Linux
HPC
Java
Fault-based testing
efficient exploration
unit
mutation operators
multithread code
execution time
Execution time
Experiment
Dynamic
Shared memory
backtrackable JVM
No Information
No Information
StateSet technique
atomicity violation, data race, deadlock
MutMut
Academic
Linux
Generating effective tests for concurrent programs via AI automated planning techniques
No Information
HPC
Java
Formal method-based testing
Search based testing
unit
predict interleavings
concurrent program
number of predicted runs, the time required for each prediction, number of bugs found during the tests for each violation type
effectiveness, performance, prediction capability
Case study
Static, Dynamic
Shared memory
scheduling
No Information
No Information
selection patterns based on violation patterns
data race, atomicity violations, null-pointer dereferences
Penelope
Academic
No Information
Generating unit tests for concurrent classes
No Information
HPC
Java
Structural testing
Search based testing
unit
concurrency coverage
bytecode of a class
number of data races found
effectiveness
Case study
Static, Dynamic
Shared memory
execution schedules
control flow graph
Instrumentation with ConSuite
meta-heuristic search
data race, deadlock
ConSuite, EvoSuite
Academic
No Information
How Good is Static Analysis at Finding Concurrency Bugs?
No Information
HPC
Java
Fault-based testing
static analysis
unit
data flow analisis, lock dependency, call-graph analysis, alias analysis, thread-scape analysis, lock analysis
java bytecode
number of bugs detected and percentage of spurious results produced by the tool
effectiveness
Survey
Static
Shared memory, Message passing
Not required
call graph, lock graph
No Information
No Information
data race, deadlock, starvation, bug patterns
FindBugs, Jlint, Chord
Academic
No Information
Location pairs: a test coverage metric for shared-memory concurrent programs
No Information
HPC
Java
Structural testing
Coverage metric, Location Pair (LP)
unit
location pairs
java bytecode
test adequacy: percentage of iterations for which a metric was successful, "% of passes that cover a new MP/DU/LP that also detect the bug” presents experimental results intended to show the correlation between the MP, DU and LP metrics and the atomicity violations in the buggy programs studied , "% of passes that detect the bug that also cover a new MP/DU/LP” is intended to provide another measure of correlation between coverage metrics and concurrency errors
effectiveness, efficiency, scalability
Case study
Static, Dynamic
Shared memory
controlled thread schedule
control flow graph
JPF instrumentator
state transition coverage
high-level concurrency errors, atomicity violation, refinement violations, data race
Monitoring tool, JavaPathfinder
Academic
No Information
The Impact of Concurrent Coverage Metrics on Testing Effectiveness
No Information
HPC
Java
Structural testing
guided for testing criteria
unit
coverage metrics (singular, pairwise)
java code
coverage achieved, test suite size, fault detection effectiveness
effectiveness
Experiment
Dynamic
Shared memory
java thread scheduling
No Information
No Information
greedy test suite reduction
mutation based errors
No Information
No Information
No Information
A Lightweight and Portable Approach to Making Concurrent Failures Reproducible
Windows
Distributed systems
Java
Structural testing
replay-based
unit
logical thread schedule
java code
effectiveness: number of generated test cases and reproducible test cases, overhead: execution time. failure reproducible
effectiveness, efficiency, performance, reproducibility
Case study
Static
Shared memory
scheduling
No Information
instrumentation schema
static data race detection
deadlock, data race, atomicity violation
Concrash
Academic
Windows
An approach to testing commercial embedded systems
No Information
Embedded systems
C
structural testing
coverage criteria, test oracle
unit
definition-use pairs
embedded system code
the numbers of faults in our object programs detected by the different test suites
effectiveness
Experiment
Dynamic
Shared memory
runtime monitoring
ICFG(Interprocedural control flow graph)
on time instrumentation
greedy test suite reduction
N
No Information
No Information
No Information
An Automation-Assisted Empirical Study on Lock Usage for Concurrent Programs
Unix
Distributed systems
Java
No Information
lock pattern
unit
symbolic exectuion
concurrent program
lock manifestation, lock pattern and lock usage evolution
lock manifestation
Experiment
Static
Shared memory
deterministic execution
control flow graph, use-define chains, control dependency graph
No Information
common locking patterns analysi
data race, deadlock
Lupa
Academic
Unix
An empirical study of data race detector tools
No Information
HPC
Java
Fault-based testing
data race detection
system
No Information
java source code, java bytecode
Compilation Time (CT) in second, Number of Line of Code (LOC), Number of Classes (NOC), Number of Reported Data Race (RDR), Number of Actual Data Race (ADR), Type of Data Race (TDR), Output Result (OR), Tool Usage (TU)
correctness, performances, effectiveness
Survey
Static, Dynamic
Shared memory
memory history
No Information
No Information
No Information
data race
RaceFuzzer, RacerAJ, JCHORD, RCC, JRF
industry
No Information
Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation
No Information
HPC
Java
Structural testing
coverage criteria
unit
concurrency coverage metrics
java code
Achieved concurrency coverage of test suites, Test suite size, Fault detection effectiveness of generated test suites, Difficulty of satisfying test requirements, Fault detection effectiveness of test requirements
coverage, cost, effectiveness, coverage
Experiment
Dynamic
Shared memory
random execution
No Information
No Information
greedy test suit eduction
mutation based errors
PCT, Ctrigger
Academic
No Information
ASP: Abstraction Subspace Partitioning for Detection of Atomicity Violations with an Empirical Study
Linux
Distributed systems
Java
Fault-based testing
partition-based
No Information
patterns of non-serializable interleaving
multithreaded program
percentage of bugs detected versus the percentage of suspicious instances confirmed
cost, effectiveness
Experiment
Dynamic
Shared memory
happens-before and locksets of memory access
No Information
No Information
subspace partitioning
atomicity violation
Maple
Academic
Linux
Assertion Guided Symbolic Execution of Multithreaded Programs
No Information
HPC
C, C++
Structural testing
slicing testing
No Information
symbolic exectuion
multithreaded program
number of explored executions, run time
performance, effectiveness, scalability
Case study
Static, Dynamic
Shared memory
symbolic execution
Generalized Interleaving Graph (GIG)
LLVM instrumentation
minimization heuristics, partial order reduction
N
Cloud9
Academic
No Information
Automated Classification of Data Races Under Both Strong and Weak Memory Models
Linux
HPC
C, C++
Fault-based testing
data race detection
unit
symbolic exectuion
multithreaded program
effectiveness: Number of Bugs finding and states coverage, accuracy: classification accuracy, Scalability: number of executions per time, efficiently: runing time memory, overhead:peak memory consumption, performance: time to classify
effectiveness, accuracy, performance
Case study
Dynamic
Shared memory
replay-based classification
happens-before graph
dynamic instrumentation
upper bound od primary paths explored, reduct branches in the execution tree
data race
Portend+
Academic
Linux
Clash of the Titans: Tools and techniques for hunting bugs in concurrent programs
No Information
HPC
java, C#
No Information
stateless random walk, randomized depth-first search, guided search using abstraction refinement
unit
context-bound
C#, C, Java code
Hardness: ratio of the total number of trials executed over he total number of error discovering trials, Time Bound: time bound at one hour, Number of Trials: number trials executed
effectiveness
Case study
Dynamic
Shared memory
scheduling
No Information
No Information
dynamic partial order reduction
Hunting bugs, data race, deadlock, atomicity violation
CalFuzzer, ConTest, CHESS, Java Pathafinder (JPF)
No Information
No Information
Concurrency Testing Using Schedule Bounding: an Empirical Study
No Information
HPC
C, C++, Java
Formal method-based testing
context bounding
No Information
iterative preemption bounding (IPB), iterative delay
multithread algoriths
schedules, time, bug found
effectiveness
Case study
Dynamic
Shared memory
systematic concurrency testin
No Information
No Information
scheduling bounding
deadlock, crashes, assertion violation, data race
Maple
Academic
Linux
ConLock: A Constraint-based Approach to Dynamic Checking on Deadlocks in Multithreaded Programs
Linux
HPC
C, C++, Java
Model based testing
constraint based
unit
bounding (IDB) and unbounded depth-first search (DFS), non-systematic testing with a naive random scheduler (Rand)
Java, C, C++ binary program
performance: time, Effectiveness on Real Deadlocks: number of real deadlock confirmed, Effectiveness on False Positives: number of false positive reported
effectiveness, performance
Case study
Dynamic
Shared memory
should-happen-before
No Information
Pin instrumented
reduction of constraints
deadlock
MagicFuzzer
Academic
Linux
Detecting Event Anomalies in Event-based Systems
Android
Mobile
Java
Fault-based testing
data flow analysis
integration
event-based interaction
android applications
execution time, number of false positives, number of false negatives
accuracy, execution time
Case study
Static
Shared memory
monitoring
control-flow graph (CFG), inter-procedural control-flow graph(ICFG)
No Information
No Information
data race
DEvA
Academic
Windows
Effective random testing of concurrent programs
No Information
HPC
Java
Model based testing
random partial order sampling
unit
happens-before relation
multithread Java programs (data structures, high-performance computing, examples of synchronization mechanisms)
number of executions required to find a defect, number of partial orders sampled
effectiveness
Case study
Dynamic
Shared memory
scheduling
No Information
CalFuzzer instrumentor
random partial order reduction
deadlock, data race, assertion violation
CalFuzzer
Academic
No Information
Empirical evaluation of a new composite approach to the coverage criteria and reachability testing of concurrent programs
Linux
HPC
C, MPI
Structural testing
Guided for testing criteria
unit
Reachability testing
concurrent code in MPI
number of faults found, number of faults injected, number of SYN-sequences (or paths) executed for each approach
efficiency, effectiveness
Experiment
Static, Dynamic
Message passing
SYN-sequences
Parallel Control Flow Graph (PCFG)
instrumentation
feasible synchronization sequences
mutation based errors
ValiMPI
Academic
Linux
Incremental integration testing of concurrent programs
No Information
Labeled transitions systems
CCS, CSP, Ada
Structural testing
coverage criteria
integration
all-paths, all-proper-paths, all-transitions, all-states, all-int-T-synchronizations, all-int-L-synchronizations
abstract and concrete concurrent programs
the number of transitions and states of a particular reduced ALTS, the number of states in the reduced ALTS as a percentage of the number of states in the unreduced ALTS, average sizes of the program slices produced using algorithm, the number of int-T-synchronizations, int-L-synchronizations in the reduced models compared the level of effort required to satisfy the all-T-synchronization, all-Lsynchronization coverage criteria, number of test paths
effectiveness, effort, state space reduction
Case study
Static, Dynamic
Message passing
incremental reachability analysis
reachability graph
No Information
partial-order reduction
interface-level synchronization
No Information
No Information
No Information
Learning from mistakes: a comprehensive study on real world concurrency bug characteristics.
No Information
Distributed systems
C, C++
Fault-based testing
bug-triggering
system
partial order
open source program
number of bugs detected
bug detection
Survey
Dynamic
Shared memory
transactional memory
No Information
No Information
exponential-sized all-interleaving testing sc
atomicity violation, order violation
No Information
No Information
No Information
RACEZ : A Lightweight and Non-Invasive Race Detection Tool for Production Applications
Linux
Distributed systems
C, C++
Fault-based testing
dynamic data race detection
unit
data flow analysis
multithread program
overhead and false positives
effectiveness, efficiency
Case study
Static
Shared memory
monitoring
No Information
lockset instrumentation
false positive reduction
data race
Racez
Academic
Linux
RECONTEST: Effective Regression Testing of Concurrent Programs
No Information
HPC
Java
Structural testing
new interleaving selection
regression
predictive trace analysis
concurrent program
number of potential violations detected by the number or violations predicted, computation time, number of violations reported by RECONTEST
effectiveness, efficiency, correctness
Case study
Dynamic
Shared memory
monitoring, lock-set, happens-before
No Information
soot instrumentation
delta of the interleaving space
faulty interleaving
Recontest
Academic
No Information
Runtime Prevention of Concurrency Related Type-state Violations in Multithreaded Applications
Linux
HPC
C, C++, Pthreads
Fault-based testing
data flow analysis
unit
delay method
concurrent program
effectiveness: supression of type-state violations, performance: overhead of runtime failure mitigation, scalability: increasing the number threads and runtime overhead
effectiveness, performance, scalability
Case study
Dynamic
Shared memory
monitoring
type-state automaton
llVM instrumentation
partial order reduction
type-state violations
LLVM framework
Academic
Linux
Saturation-based testing of concurrent programs
No Information
HPC
Java
Structural testing
coverage metrics, saturation-based criteria
unit
randomized scheduler
concurrent program
coverage values
cost, effectiveness
Case study
Static, Dynamic
shared memory
deterministic execution
No Information
No Information
No Information
N
No Information
No Information
No Information
Selective mutation testing for concurrent code
Linux
HPC
Java
Fault-based testing
selective mutation
unit
selective mutation
java code
time to execute schedules, number of generated mutants, number of killed mutants, number of hit mutants, mutation scores
cost, effectiveness
Experiment
Dynamic
Shared memory
scheduling
No Information
No Information
selective mutation
concurrency bugs
Comutation
Academic
Linux
Testing Concurrent Programs on Relaxed Memory Models
No Information
HPC
C
Structural testing
random scheduler
unit
noise making
C program
Average Runtime (seconds), cycles predicted, cycles confirmed, number of bugs identified, probability of confirming a cycle
feasibility
Case study
Dynamic
Shared memory
controlled scheduling
No Information
CIL instrumentation
No Information
data race
Relaxer
Academic
No Information
Toward a Methodology to Expose Partially Fixed Concurrency Bugs in Modified Multithreaded Programs
No Information
HPC
Pthread
Structural testing
similarity-based active testing strategy
regression
similar interleaving
multithreaded program
Performance Maple: the time needed in both the profiling and testing iterations needed to trigger the same bug in each modified version, Performance RegressionMaple: time needed for the testing iterations needed to trigger the bug
performance
Case study
Dynamic
Shared memory
regression scheduler
No Information
No Information
call stack similarity
deadlock, data race
RegressionMaple
Academic
No Information
Incremental testing of concurrent programs using value schedules
Windows
HPC
Java
Model based testing
read-write assignment sequences
unit
value-schedule-based technique
java programs
number of transitions and intermediate states required to cover all relevant behaviors of the program or discover a bug
functionality, applicability
Case study
Static, Dynamic
Shared memory
scheduling
control-flow graph
soot instrumentation
partial order reduction
deadlock, data race
JPF
Academic
Windows
Efficient Deterministic Multithreading Without Global Barriers
Linux
HPC
C, C++, Pthreads
Model based testing
deterministic lazy release consistency (DLRC)
unit
deterministic lazy release consistency memory model
parallel applications
number of synchronization operations, number of memory operations, the memory footprint
accuracy, execution time
Case study
Static, Dynamic
Shared memory
record and replay
deterministic lazy release consistency model
RFDet compile-time instrumentation
happens-before slices
data race
RFDet
Academic
Linux
Efficient Execution Path Exploration for Detecting Races in Concurrent Programs
No Information
Distributed systems
Java
Structural testing
read-write sequences
unit
Reachability testing
concurrent programs
number of test cases, number of interleavings
effectiveness
Case study
Dynamic
Shared memory
deterministic replay
variant graph
Aspectj tracing
partial order reduction
data race
AspectJ
Academic
No Information
Pegasus: Automatic Barrier Inference for Stable Multithreaded Systems
Linux
HPC
C, C++
Structural testing
Automatic Barrier Inference
system
soft barriers
cuncurrent program source code
execution time
effectiveness, performance
Case study
Dynamic
Message passing
semantic order
constraint graph
instumentation generated by PARROT
conditional barriers
bottlenecks
Pegasus
Academic
Linux
Race directed random testing of concurrent programs
Sun
HPC
Java
Fault-based testing
randomized dynamic analysis
system
randomized thread scheduler
the initial state and a set of two racing statements
average runtime, number of potential races detected, number of real races reported, number of exceptions
effectiveness
Case study
Dynamic
Shared memory, Message passing
happens-before relation
No Information
Racfuzzer instumentation
minimal annotations
data race, exeption
RaceFuzzer
Academic
Sun
SATCheck: SAT-Directed Stateless Model Checking for SC and TSO
Linux
HPC
C
Formal method-based testing
model checking
unit
SAT-based approach
C code
execution time
performance
Case study
Dynamic
Shared memory
SAT solving
event graph
Instrumentation with satcheck
partial order reduction
N
SATCheck
Academic
Linux
TRADE: Precise Dynamic Race Detection for Scalable Transactional Memory Systems
No Information
Transactional memory
C, C++
Fault-based testing
read-write sequences
system
relaxed happens-before relation
SMT program
number of existing and injected data races detected by tools, ratio of the execution time, runtime overhead
effectiveness, performance
Case study
Dynamic
Shared memory
happens-before relation
transactional memory
Instrumentation of transactional accesses
No Information
data race
TRADE
Academic
No Information
A combinatorial testing strategy for concurrent programs
No Information
HPC
Java
Structural testing
Guided for testing criteria
unit
Reachability testing
concurrent program
number of mutants alive
effectiveness, adequacy
Case study
Dynamic
Message passing
SYN-sequences
extended control-flow graph (ECFG)
No Information
t-way reachability testing strategy
synchronization bugs
RichTest
Academic
No Information
A meta heuristic for effectively detecting concurrency errors
No Information
HPC
Java
Structural testing
Search based testing
unit
meta heuristic
java code
density of error discovery
effectiveness
Case study
Static
Shared memory
deterministic execution
AbsList, AryList, Deadlock models
Jlint instrumentation
partial order reduction
real errors
Jlint
Open source
No Information
A model-free and state-cover testing scheme for semaphore-based and shared-memory concurrent programs
Windows
HPC
Java
Structural testing
state-cover testing
unit
Reachability testing
java bytecode
deadlocks detected, time, max memory usage, number of states, number of threads used
feasibility
Case study
Dynamic
Shared memory
SYN-sequences
reachable state graph
No Information
on-the-fly partial order reduction
synchronization bugs
JPF
Open source
Windows
A platform for search-based testing of concurrent software
No Information
HPC
Java
Structural testing
Search based testing
unit
heuristical noise injection
concurrent program
code coverage
bug detection
Case study
Static, Dynamic
Shared memory
executor plug-in of previously explored state space
No Information
Search-Bestie instrumentation
state space reduction
synchronization bugs, exeption
Search Bestie, ConTest
Open source
No Information
A race-detection and flipping algorithm for automated testing of multi-threaded programs
Windows
HPC
Java
Structural testing
symbolic execution
unit
Reachability testing
multi-thread program
Run time in seconds, number of Paths, number of races/deadlocks/infinite loops/exceptions, numer of threads, number of functions tested
effectiveness
Case study
Static
Shared memory
concolic execution
No Information
Jcute trace
partial order reduction
data race, deadlock
jCUTE
No Information
Windows
A randomized scheduler with probabilistic guarantees of finding bugs
Windows
Distributed systems
C, C++
Structural testing
randomized scheduler
system
randomized scheduler
C/C++ executable compiled programs
percentage of bugs detected
effectiveness
Case study
Dynamic
Shared memory
scheduling
No Information
PCT binary instrumentation
probabilistic bound
atomicity violation, deadlock, preemption
PCT
No Information
Windows
Accurate and Efficient Runtime Detection of Atomicity Errors in Concurrent Programs
Unix
HPC
Java
Fault-based testing
detect atomicity violation
system
serializability
concurrent program
running time
performance, accuracy
Case study
Dynamic
Shared memory
happens-before relation
access tree
pretty-print instrumentation
reduction-based algoritm
atomicity, data race, serializability
No Information
No Information
No Information
Advances in noise-based testing of concurrent software
Linux
HPC
C, Java
Structural testing
search based testing
unit
noise injection
Java bytecode and C/C++ binaries
time, number of test cases
efficiency of detecting concurrency-related errors
Survey
Dynamic
Shared memory
deterministic execution
No Information
Contest instrumentation
partial order reduction
noise problem
No Information
No Information
No Information
An Efficient and Flexible Deterministic Framework for Multithreaded Programs
Linux
HPC
Pthread
Structural testing
Slicing based testing
system
deterministic multithreading
multithreaded programs
execution time, memory overhead, disk space overhead
performance
Case study
Dynamic
Shared memory
deterministic replay
No Information
Kendo instrumentation
slice merging
data race, synchronization
No Information
No Information
No Information
An Evaluation of Data Race Detectors Using Bug Repositories
Windows, Unix
Distributed systems
Java
Fault-based testing
data race detection
unit
No Information
multithreaded applications
bugs detected
efficiency
Case study
Static, Dynamic
Shared memory, Message passing
happens-before
No Information
MTRAT instrumentation
partial order reduction
data race, atomicity, bad optimization
MTRAT, ConTest, Jinx, Jchord, Enriched PUT
No Information
Windows, Unix
ASN: A Dynamic Barrier-Based Approach to Confirmation of Deadlocks from Warnings for Large-Scale Multithreaded Programs
Linux
Distributed systems
C, C++, Java
Fault-based testing
dynamic barrier-based approach
system
randomized scheduler
multithreaded program
deadlocks detected
effectiveness
Case study
Static, Dynamic
Message passing
deterministic replay
No Information
Pin instrumented
ASN
deadlock
No Information
No Information
No Information
Automatic testing environment for multi-core embedded software - ATEMES
Linux
Embedded systems
C, C++
Structural testing
automatic test data generation
unit
CppUnit-base test case
multithreaded program
line coverage, response time, branch coverage, executing time, failed test case data
performance, reliability, usability
Survey
Dynamic
Shared memory
monitoring
No Information
ATEMES instrumentor
TBB pipeline parallelism dimension
data race
ATEMES, Intel TBB
No Information
Linux
Bita: Coverage-guided, automatic testing of actor programs
No Information
actor programs
Scala
Structural testing
Guided for testing criteria
system
schedule coverage criteria
actor systems
number of bugs detected, number of schedules generated, number of executions until detect the bug, average time to find a bug.
effectiveness
Case study
Static, Dynamic
Shared memory, Message passing
scheduling
No Information
Bita instrumentation
dynamic partial order reduction
bugs inserted by the researcher
No Information
No Information
No Information
CLAP: Recording Local Executions to Reproduce Concurrency Failures
Linux
Distributed systems
C, C++
Fault-based testing
constraint based
system
preemption-bounded schedule generation
concurrent applications
Runtime and space overhead, number of schecules, number of bugs rerproduced
effectiveness, performance
Case study
Static
Shared memory
symbolic execution
TSO/PSO models
CLAP instrumentation
preemption-bounding
data race
LLVM, KLEE 2.9, STP, LEAP
No Information
No Information
Composable Specifications for Structured Shared-Memory Communication
Linux
HPC
Java
Formal method-based testing
composable specifications
unit
communication inlining
java applications
size of anotation, number of specified or exercised communication, runtime overhead (time and space), number or anotatition.
accuracy, performance
Case study
Dynamic
Shared memory, Message passing
No Information
No Information
JIT instrumentation
omitting inlined methods
code-communication invariants
OshaJava
No Information
Linux
Con2Colic Testing
Linux
HPC
Java
No Information
concolic testing
unit
context-bound
concurrent code
diagnosis time, number of iterations required to complete the diagnosis, average size of the root cause returned by our method,
effectiveness, scalability
Case study
Static
Shared memory
symbolic execution
interference scenarios
Concrest instrumentation
con2colic
data race, bugs inserted by the researcher
ConCrest
No Information
Linux
ConcBugAssist: Constraint Solving for Diagnosis and Repair of Concurrency Bugs
Linux
Distributed systems
C
Formal method-based testing
bounded model checking
system
logical constraint based symbolic analysis
multithreaded program
reduction ratio, i.e., the number of constraints in the root cause divided by the average number of constraints in a bad schedule
effectiveness
Case study
Dynamic
Shared memory
happens-before constraints
constraint model
CBMC traces
minimally unsatisfiable subformula
assertion violation, data race, order violation
CBMC, MSUnCore, ConcBugAssist
Academic
Linux
Concurrency Debugging with Differential Schedule Projections
Linux, Unix
HPC
C, C++, Java
Model based testing
concolic testing
system
symbolic constraint solver
multithreaded program
efficiency: time and storage overhead of path profiling relative to native execution and the time cost of symbolic trace collection, efficacy: the number of program events and data-flow edges in the full, failing execution that Symbiosis computes compared to the number of events and data-flow edges in the differential schedule projection
efficiency, efficacy
Case study
Static
Shared memory
concolic execution
constraint model
LLVM instrumentation
Differential schedule projections
data race, atomicy violation, bugs inserted by the researcher
LLVM, Soot
Academic
Linux, Unix
Concurrent software verification with states, events, and deadlocks
Linux
Distributed systems
C
Formal method-based testing
model checking
system
state/event verification
C program
deadlock: maximum number of states, number of reachable states, number of iterations, time in seconds, memory usage in MB, effectiveness: number of states and transitions in the the model, model construction time, model checking time, total verification time
effectiveness
Case study
Dynamic
Message passing
automata-based specifications
LKS labeled kripke structures
No Information
abstractions and compositional reasoning
deadlock, SO bugs
MAGIC
Academic
Linux
ConMem: Detecting severe concurrency bugs through an effect-oriented approach
Linux
Distributed systems
C, C++
Fault-based testing
effect-oriented approach
unit
controlled interleaving perturbation
concorrent program with bugs
effectiveness: number of bugs detected, performance: average time across multiple runs, accuracy:numbers of false positives (vs. true bugs)
effectiveness
Case study
Static
Shared memory
deterministic replay
No Information
PIN instrumentation
effect-oriented approach
concurrency bugs
ConMem
Academic
Linux
ConSeq: Detecting Concurrency Bugs through Sequential Errors
Linux
HPC
C, C++
Fault-based testing
state-space search
unit
consequence oriented approach
concurrent applications
number of bugs detected, number of false positives, runtime overhead
effectiveness, accuracy, performance
Case study
Dynamic
Shared memory
symbolic execution
program dependence graphs
Pin instrumented
suspicious interleaving
data race, atomicity violations
ConSeq
Academic
Linux
Coverage guided systematic concurrency testing
Linux
HPC
C, C++, Pthreads
No Information
coverage criteria
unit
coverage guided selective search
real-world concurrent program
performance: number of interleavings tested and the run time in seconds, effectiveness: if a bug is found
performance, effectiveness
Case study
Dynamic
Shared memory
control of a scheduler process
No Information
Fusion instrumentation
Partial order reduction, History-aware Predecessor-Set (HaPSet)
deadlock, atomicity violation, order violation
Fusion
Academic
Linux
Coverage metrics for saturation-based and search-based testing of concurrent software
Linux
Distributed systems
Java
Structural testing
Search based testing
unit
synchronization coverage
multi-threaded java programs
percentage of code coverage
accuracy
Case study
Static, Dynamic
Shared memory
interleaving scenarios
guarded lock graph
Contest instrumentation
saturation-based
data race, deadlock, synchronization, atomicity violation
No Information
No Information
Linux
CTrigger: Exposing atomicity violation bugs from their hiding places
No Information
Distributed systems
C, C++
Fault-based testing
detect atomicity violation
system
bug-triggering interleaving
server/desktop application, concurrent applications
effectiveness: whether the bugs can be exposed, efficiency: how quickly the bugs can be exposed, reproducibility: how reliably the bugs can be reproduced after their first manifestation
effectiveness, efficiency, reproducibility
Case study
Static
Shared memory
controlled testing
No Information
Pin instrumented
infeasible interleaving prunning
atomicity violation
Ctrigger
Academic
No Information
Distributed reachability testing of concurrent programs
Linux
Distributed systems
Java
Model based testing
SYN-sequences
system
reachability testing
java programs
speedup
performance
Case study
Dynamic
Shared memory, Message passing
prefix-based execution
lts model program
No Information
t-way reachability testing strategy
synchronization bugs, data race
RichTest
Academic
Linux
Effective and Precise Dynamic Detection of Hidden Races for Java Programs
Unix
HPC
Lotus, Java
Fault-based testing
delay scheduler
unit
may-trigger relation
real world java multithreaded program
number of races detected, overhead time, Thrashing Rate
effectiveness, performance, thrashing Rate
Case study
Dynamic
Shared memory
happens-before relation
prediction model
DrFinder instrumentation
active scheduling
Hidden races, synchronization bugs
DrFinder
Academic
Unix
Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool
Linux
Embedded systems
C, C++, Pthreads
Formal method-based testing
embedding semantics
unit
SPSC bounded lock-free queue
parallel applications
number or data races detected
bug detection
Case study
Dynamic
Shared memory
happens-before
No Information
LLVM instrumentation
lock free structures
data race
ThreadSanitizer, LLVM
Academic
Linux
Enforcer - Efficient failure injection.
Unix
HPC
Java
Structural testing
guided for testing criteria
unit
all-e-deacts criterion
java programs
exception coverage after Enforcer was used
usefulness
Case study
Static, Dynamic
Not informed
Enforcer run-time library
No Information
Enforcer instrumentation
coverage metrics
bugs inserted by the researcher
Enforcer, Junit
No Information
Unix
Exploiting parallelism in deterministic shared memory multiprocessing
Linux
HPC
Java
Model based testing
working set memory
system
Semantics-aware determinism
multi-threaded java programs
time, scalability, redistribution granularity, maximum execution budget
determinism, performance, effectiveness
Case study
Dynamic
Shared memory
happens-before
weak memory consistency models
LLVM instrumentation
redistribution granularity
data race, concurrency bugs
LLVM, FPDet
No Information
Linux
Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing
No Information
Distributed systems
C, C++
Fault-based testing
Guided for testing criteria
system
data flow criteria
C/C++ multithreaded applications
Bug-Exposing Time, CTrigger Bug-Exposing Time Breakdown, Unserializable Interleaving Coverage
effectiveness, efficiency, reproducibility
Case study
Static
Shared memory
bug-triggering
No Information
Pin instrumentation
Pruning infeasible interleavings
atomicity violation
PIN
No Information
No Information
Iterative context bounding for systematic testing of multithreaded programs
Windows
HPC
C, C++, C#
Formal method-based testing
model checking
system
context bounding
multithread code
number of preemptions, number or executions
efficacy
Case study
Static
Shared memory, Message passing
happens-before relation
coverage graph
No Information
iterative context-bounding
data race, synchronization
Zing, CHESS
No Information
Windows
Kendo: efficient deterministic multithreading in software
Linux
HPC
C, C++
Structural testing
deterministic logical time
system
deterministic multithreading
parallel applications
percentage of nondeterministic (pthreads) execution time
performance
Case study
Static, Dynamic
Shared memory
deterministic logical time
deterministic multithreading programming models
No Information
No Information
data race, deadlocks, atomicity violation, order violation
Kendo
No Information
Linux
Kivati: Fast detection and prevention of atomicity violations
Linux
Distributed systems
Pthread, C
Fault-based testing
guided for testing criteria
system
data flow criteria, local memory access pairs
C program
number of bugs detected, number of false positives, run-time overhead
effectiveness, performance
Case study
Static
Shared memory
hardware wachpoint register
No Information
thead trace
path-insensitive data-flow analysis (DFA)
atomicity violation
Kivati
Academic
Linux
Maple: a coverage-driven testing tool for multithreaded programs
Linux
HPC
C, C++
Structural testing
coverage-driven approach
unit
active scheduling
multithreaded program
average performance overhead, Number of successfully exposed iRoots/number total predicted iRoots
efficiency, effectiveness
Case study
Dynamic
Shared memory
Thread interleaving orchestration
No Information
Pin instrumentation
iRoots coverage
data race, atomicity violation, deadlock
Maple
Open source
Linux
Maximizing synchronization coverage via controlling thread schedule
No Information
HPC
Pthread
Structural testing
code coverage analysis
unit
active scheduling
multi-threaded applications
time to achieve full coverage
effectiveness
Case study
Dynamic
Shared memory
controlled thread schedule
No Information
DynInst instrumentation
No Information
synchronization bugs
DynInst
No Information
No Information
Multicore Acceleration of Priority-Based Schedulers for Concurrency Bug Detection
Linux
HPC
C++
Structural testing
priority-based probabilistic concurrency testing
unit
PPCT randomized scheduler
C++ code
Runs with the bug found, execution time overhead
effectiveness, performance
Case study
Dynamic
Shared memory
controlled thread schedule
No Information
Pin instrumentation
preemption-bounding
atomicity violation, deadlock, order violation
NeedlePoint
Open source
Linux
MultiOtter : Multiprocess Symbolic Execution
Unix
Distributed systems
C
Model based testing
symbolic execution
system
syn-sequences coverage
server programs
line coverage, number of paths explored
performance
Case study
Dynamic
Shared memory
scheduling
No Information
No Information
grammar-contrained input
atomicity violation
MultiOtter
No Information
Unix
Multithreaded Test Synthesis for Deadlock Detection
Linux
HPC
Java
Fault-based testing
randomized test case generation
unit
synthesized tests
java bytecode
number of deadlocks syntetized, numbers corresponding to the synthesized multithreaded tests, the deadlocks detected
effectiveness
Case study
Dynamic
Shared memory
lock dependency relation
object-sensitive lock graph
soot instrumentation
constraint-based
deadlock
OMEN
No Information
Linux
On a Technique for Transparently Empowering Classical Compiler Optimizations on Multithreaded Code
Linux
HPC
C, C++
Structural testing
Guided for testing criteria
unit
data flow analysis
multithreaded program
compilation time
performance
Case study
Static
Shared memory
siloed references
PCG graph
No Information
concurrency-type refinaments
data race, synchronization bugs
No Information
No Information
No Information
Race Detection for Android Applications
Android
Mobile
Java
Fault-based testing
Event driven programming model
system
control flow
android applications
number of data races reported, number of reports generated by DROIDRACER, number of true and false positives
performance
Case study
Dynamic
Shared memory
happens-before relation
happens-before graph
Droidracer instrumentation
android runtime enviroments to reduce false positives
data race
DroidRacer
No Information
Android
Reachability testing of concurrent programs.
Windows, Solaris, Unix
Distributed systems
C++
No Information
No Information
system
Reachability testing
multithreaded program
number of sequences exercised during reachability testing, number of times the programs are executed, the number of transitions that are explored, SYN-sequences explored
performance, effectiveness
Case study
Static
Message passing
prefix-based execution
graph-theoretic
No Information
partial order reduction
data race, synchronization bugs
RichTest
No Information
Windows, Solaris, Unix
Runtime analysis of atomicity for multithreaded programs.
Solaris
HPC
Java
Fault-based testing
block-based, reduction-based algorithm
unit
synchronization events
multithreaded programs
atomicity violations reported, the number of missed errors for the online reduction algorithm, running times
usability, accuracy, performance
Case study
Dynamic
Shared memory
runtime monitoring
happens-before graph
Kopi instrumentation
commutativity properties of events
atomicity violation
No Information
No Information
No Information
SOS: Saving Time in Dynamic Race Detection with Stationary Analysis
No Information
HPC
Java
Fault-based testing
data race detection
system
stationary objects
multithreaded applications
runtime, race detected, overhead
performance, effectiveness
Case study
Dynamic
Shared memory
monitoring lock usage
No Information
Jikes RVM instrumentation
stationary-object optimization
data race
Jikes RVM 3.1.0
No Information
No Information
Test-Data Generation for Testing Parallel Real-Time Systems
Unix, Linux
HPC
C
Functional testing
Search based testing
system
worst case execution time
real-time systems
end-to-end execution time
performance, scalability
Case study
Static
Shared memory
monitoring
No Information
FastTrack instrumentation
GA algorithm
bugs inserted by the researcher
No Information
No Information
Linux
TM UNIT : A Transactional Memory Unit Testing and Workload Generation Tool
Linux
Transactional memory
Pthread, C
Formal method-based testing
model checking
unit
variable and thread-bounding
lock-based applications
overhead of the dynamic execution
performance
Case study
Static
Shared memory
recorded traces
automaton
Valgrind instrumentation
No Information
bugs inserted by the researcher
Tmunit
No Information
Linux
Variable and thread bounding for systematic testing of multithreaded programs
No Information
HPC
Java, C#
Formal method-based testing
model checking
unit
variable and thread-bounding
Java and C# programs
running time, average number of executions, time required to capture a bug
performance, effectiveness
Case study
Static
Shared memory
record-replay
happens-before graph
Javassist library instrumentation
variable and thread ranking
bugs inserted by the researcher
RankChecker
Academic
No Information
Applications of Model Reuse When Using Estimation of Distribution Algorithms to Test Concurrent Software
No Information
Distributed systems
Java
Model based testing
Search based testing
system
model reuse, estimation distributed algorithm EDA
system specification in LTL
time in finding a error
efficiency
Case study
Dynamic
shared memory
model reuse
state space graph, ltl model
No Information
fitness functions, heuristic mechanisms
deadlock, bugs inserted by the researcher
ECJ, HSF-SPIN
Academic
No Information
Value-deterministic Search-based Replay for Android Multithreaded Applications
Linux
Mobile
Java
Structural testing
Search based testing
system
order-based technique
android applications
execution time
efficiency
Case study
Dynamic
Shared memory, Message passing
No Information
No Information
Aspectj instrumentation
value-deterministic search-based replay
data race
Dalvik Debug Monitor Server (DDMS)
No Information
Linux
On the Effectiveness of Contracts as Test Oracles in the Detection and Diagnosis of Functional Faults in Concurrent Object-Orien
No Information
Distributed systems
Java
Functional testing, Formal method-based testing
designing contracts
system
runtime assertion checking
concurrent object-oriented software
probability of oracles detecting faults, average effort to diagnose faults normalized with the size of an execution thread
effectiveness
Case study
Dynamic
Message passing
No Information
system specification
RAC instrumentation
No Information
data race, deadlock
JML toolset
No Information
No Information
Types for safe locking: Static race detection for Java
Unix
Distributed systems
Java
Formal method-based testing
annotation inference
system
candidates annotations
unannotated program
number of annotations and time required to annotate each program, the number of race conditions found in each program, the time spent by the programmer inserting annotations, the time to run the tool
effectiveness
Case study
Static
Shared memory
lock-based synchronization
No Information
No Information
No Information
data race
RaceFreeJava
No Information
Unix
AVIO: Detecting atomicity violations via access interleaving invariants
No Information
Distributed systems
C, C++
Fault-based testing
concurrent-program invariant
system
access interleaving invariants
server applications
number of bugs detected, number of false positives
effectiveness
Case study
Dynamic
Shared memory
happens-before
No Information
Pin instrumentation
No Information
Aaomicity violation
AVIO-S, AVIO-H
Academic
No Information
Detection of asynchronous message passing errors using static analysis
LInux
Distributed systems
Erlang
Structural testing
interprocess communication topology
unit
control flow analysis
Erlang programs
Number of message passing problems identified by the analysis, elapsed wall clock time (in seconds), memory requirements (in MB)
effectiveness, performance
Case study
Static
Message passing
static analysis
control-flow graph
No Information
control flow analysis
synchronization bugs
Dialyzer
No Information
Linux
DTAM: Dynamic Taint Analysis of Multi-threaded Programs for Relevancy Malay Ganai.
Linux
HPC
C, C++
Structural testing
DTAM: Dynamic Taint Analysis
system
alternate thread interleaving
multi-threaded applications
ratio of relevant inputs to the total number of inputs
relevancy
Case study
Dynamic
Shared memory
monitoring
No Information
Pin instrumentation
relevancy analysis
data race specific bugs
DTAM
Academic
Linux
Efficient concurrency-bug detection across inputs
Linux
HPC
C, C++
Structural testing
Concurrent Function Pairs (CFP)
system
concurrent function pairs
multi-threaded software
number of inputs that report each bug, number of selected inputs and functions, number of races, single-variable atomicity violations overlap across inputs, trace size, trace-analysis time
effectiveness, performance, false negatives
Case study
Static
Shared memory
lock-set, happens-before
call graph
LLVM instrumentation
CFP-guided approach
data race, atomicity violations
PIN, LLVM
No Information
Linux
Efficient data race detection for distributed memory parallel programs
Cray XT4 system
Distributed systems
C, UPC
Fault-based testing
active testing
unit
predictive analysis
distributed memory programs
runtime, overhead of the program with race prediction enabled and the average runtime, overhead for program re-execution with race confirmation (phase II), The total number of potential races, Speedup
bug detection, scalability
Case study
Dynamic
Shared memory
controlled execution
skip lists
UPC instrumentation
active testing
data race
Calfuzzer
No Information
No Information
InstantCheck: Checking the determinism of parallel programs using on-the-fly incremental hashing
No Information
HPC
C++
Structural testing
incremental rashing
system
on-the-fly incremental hashing
parallel applications
overhead, detected bug, number of determinism checks
performance, determinism
Case study
Dynamic
Shared memory
deterministic replay
No Information
Pin instrumentation
state equatily check
data race, atomicity violation, deadlock
InstantCheck
No Information
No Information
MuTMuT: Efficient Exploration for Mutation Testing of Multithreaded Code.
Linux, Sun
HPC
Java
Fault-based testing
mutation operators
unit
systematic exploration of possible schedules
multithreaded program
running time
performance
Experiment
Dynamic
Shared memory
backtrackable Java
state space graph
No Information
partial-order reduction
data race, atomicity violation, deadlock
MutMut
Academic
Linux
SAM: Self-adaptive dynamic analysis for multithreaded programs
Windows
HPC
Java
Structural testing
intraprocedural analysis, symbolic context
Integration
data flow ananalysis, thread-based escape analysis
multithreaded java programs
run time, number of redundant contexts, number of total context checks performed, atomicity violation coverage
performance, effectiveness
Case study
Dynamic
Shared memory
monitoring
No Information
compile-time and run-time instrumentation
avoid monitoring repeated events
data race, atomicity violations
SAM
Academic
Windows
Sound Predictive Race Detection in Polynomial Time
Windows
Distributed systems
Java
Fault-based testing
lock-set based approach
system
causally-predeces relation
server level applications
number of races detected, running time
performance, prediction capability
Case study
Dynamic
Shared memory
predictive analysis
No Information
RoadRunner instrumentation
sound predictive
data race
No Information
No Information
No Information
Synchronization identification through on-the-fly test
No Information
HPC
Pthread
Structural testing
synchronization pair
system
modularized and ad-hoc synchronization
multi-threaded programs
identified sync pairs, number of false positives, number of testing threads, Testing Points, running time of SyncTester, throughput
effectiveness, efficiency, execution time
Case study
Dynamic
Shared memory
monitoring
No Information
Pin instrumented
sync pairs
data race, synchronization
SyncTester
No Information
No Information
Synthesizing Racy Tests
Linux
HPC
Java
Fault-based testing
synthesizing multithreaded tests
unit
racing schedules
open-source Java libraries
races detected, synthesized test count, synthesis time
effectiveness, efficacy
Case study
Dynamic
Shared memory
synchronization traces
No Information
Soot instrumentation
synthesizing racy
data race
Narada, RaceFuzzer
Academic
Linux
Synthesizing Tests for Detecting Atomicity Violations
Linux
HPC
Java
Fault-based testing
synthesizing multithreaded tests
unit
racing schedules
open-source Java libraries
number of synthesized tests, number of true atomicity violations, time
effectiveness
Case study
Dynamic
Shared memory
synchronization traces
No Information
Soot instrumentation
data flow constraints
atomicity violation
Intruder
Academic
Linux
TransDPOR: A novel dynamic partial-order reduction technique for testing actor programs
Unix
Distributed systems
ActorFoundry
Formal method-based testing
stateless DPOR
system
dependency relation
actor program
the number of paths executed in their entirety while exploring the specified subjects, the total number of transitions executed (across all execution paths), the total exploration time in seconds, memory usage (megabytes)
effectiveness
Case study
Dynamic
Message passing
happens-before, actor clocks
state space graph
No Information
partial order reduction
data race, deadlock
TransDPOR, Basset
Open source
Unix
Unfolding based automated testing of multithreaded programs
No Information
HPC
Java
Model based testing
No Information
unit
dynamic symbolic execution, partial order reductions
multithreaded programs
number of tests needed to fully cover the benchmarks and execution time
effectiveness, performance
Case study
Dynamic
Shared memory
scheduling point
Petri nets
soot instrumentation
dynamic partial order reduction
data race, deadlock
Unamed prototype java
Academic
No Information
Using SPIN for automated debugging of infinite executions of Java programs
No Information
Distributed systems
Java
Formal method-based testing
model checking
unit
abstraction approaches
java programs
hash projection, number or transitions, time, number of executions, number of states
efficience
Case study
Dynamic
Shared memory
monitoring
Buchi automaton
Java debug interface
runtime monitoring
deadlock, bugs inserted by the researcher
TJT
Academic
No Information
Fair stateless model checking
Windows
HPC
C++
Formal method-based testing
Fair Stateless Model Checking
unit
fair scheduling
shared-memory multithreaded programs
Number of states visited for the context-bounded, depth-first strategies both with and without fairness
efficacy, performance, effectiveness
Case study
Static
Shared memory
scheduling
priority graph
No Information
partial-order reduction
deadlock, bugs inserted by the researcher
CHESS
No Information
Windows
Verification and Coverage of Message Passing Multicore Applications
Linux
Distributed system
MCAPI
Structural testing
dynamic predictive verification
unit
mutation operators
message passing programs
running time (seconds), memory usage (megabytes)
performance
Case study
Dynamic
Shared memory, Message passing
scheduling
directed graph
Inspect instrumentation
partial order trace
deadlock, data race, assertion violation
MCAPI
No Information
Linux
Preemption sealing for efficient concurrency testing
Windows
HPC
.Net
Formal method-based testing
preemption-bounded approach
unit
preempition sealing scheduler
industrial-scale concurrent programs
number of executions explored during testing, time, Speedup
efficacy, efficiency
Case study
Dynamic
Shared memory, Message passing
preemption sealing
No Information
chess instrumentation
preemption sealing
atomicity violation, livelock, deadlock
CHESS
No Information
Windows
Concurrent program verification with lazy sequentialization and interval analysis
Linux
HPC
C
Structural testing
No Information
Unit
No Information
Multithread code
Non-deterministic sequential C program
No Information
Case study
Static
Shared memory
No Information
No Information
No Information
No Information
Bug-triggering interleavings
Lazy-CSeq
Academic
Linux
SPDebugger: A Fine-Grained Deterministic Debugger for Concurrency Code
Windows
HPC
Java
Structural testing
No Information
Unit
No Information
Concurrent code
Number of test cases to cover a criterion
Effectiveness, Cost
Experiment
Static
Shared memory
No Information
No Information
No Information
No Information
Interleavings
ASM
Open source
Windows
Effectively manifesting concurrency bugs in android apps
Unix
Mobile
Android
Functional testing
No Information
System
No Information
Apk file
Number of generated events
Effectiveness, Execution time
Case study
Dynamic, Static
Shared memory
No Information
No Information
No Information
No Information
Data race, atomicity violation
AATT prototype
Academic
Unix
DFTracker: detecting double-fetch bugs by multi-taint parallel tracking
Linux, windows
HPC
C, C++
Structural testing
Coverage criteria
Unit
No Information
Multithread code
Number of test cases to cover a criterion
Effectiveness, Efficiency, Scalability
Case study
Static
Shared memory
No Information
Control flow graph
No Information
No Information
Data race, double-fetch bugs
No Information
Academic
No Information
RaceTest: harmful data race detection based on testing technology in WS-BPEL
Windows
HPC
Java, C++
Structural testing
Coverage criteria
Unit
No Information
Concurrent code
Number of test cases to cover a criterion
Effectiveness, Efficiency
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Interleavings, Data race
No Information
Academic
No Information
Testing Multi-Threaded Applications Using Answer Set Programming
Linux, windows
HPC
Java
Structural testing
Coverage criteria
Unit
No Information
Multithread code
Modular sequences
Effectiveness, Efficiency
Case study
Dynamic, Static
Shared memory
No Information
Data flow
No Information
No Information
Interleavings, Data race, atomicity violation
No Information
Academic
No Information
Dynamic testing for deadlocks via constraints
Unix
HPC
Java, C, C++
Structural testing
Coverage criteria
Unit
Happens-before relation
Concurrent code
Number of test cases to cover a criterion
Effectiveness
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Deadlock
No Information
Academic
No Information
Concurrency Bug Avoiding Based on Optimized Software Transactional Memory
Linux
HPC
C, C++
Functional testing
Coverage criteria
System
No Information
Multithread code
Number of test cases to cover a criterion
Effectiveness, Efficiency
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Deadlock, Data race, atomicity violation
Convoider
Academic
Linux
Thread-local concurrency: a technique to handle data race detection at programming model abstraction
Unix
HPC
MPI
Structural testing
Thread interaction models
Unit
Happens-before relation
Concurrent code in MPI
Number of test cases to cover a criterion
Effectiveness
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Data race
ThreadSanitizer
Academic
Unix
A Splitting Strategy for Testing Concurrent Programs
Windows
HPC
Java
Fault-based testing
Coverage criteria
Unit
Reachability testing
Multithread code
Partially-ordered sequences
Effectiveness, Performance
Case study
Static
Message passing
No Information
No Information
No Information
No Information
Mutation based errors
SplitRichTest
Academic
Windows
Using Delta Debugging to Minimize Stress Tests for Concurrent Data Structures
Windows
HPC
Java
Structural testing
Thread interaction models
Unit
No Information
Multithread code
Number of test cases to cover a criterion
Effectiveness, Efficiency
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Bugs inserted by the researcher
TestMinimizer
Academic
Windows
Efficient scalable thread-safety-violation detection: finding thousands of concurrency bugs during testing
Windows
HPC
.Net, C#
Structural testing
Thread interaction models
Unit
Happens-before relation
Multithread code
Number of test cases to cover a criterion
Effectiveness, Efficiency
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Thread-safety violations, Data race
TSVD
Academic
Windows
Efficient verification of concurrent systems using synchronisation analysis and SAT/SMT solving
No Information
HPC
No Information
Structural testing
Coverage criteria
Unit
Reachability testing
Multithread code
Partially-ordered sequences
Effectiveness
Case study
Static
Message passing
No Information
Data flow
No Information
Incremental reachability analysis
Deadlock
DeadlOx
Academic
No Information
AVPredictor: Comprehensive prediction and detection of atomicity violations
Linux
HPC
C, C++
Structural testing
Coverage criteria
Unit
No Information
Concurrent code
Number of test cases to cover a criterion
Effectiveness, Efficiency
Case study
Dynamic
Shared memory
No Information
No Information
No Information
No Information
Atomicity violation, Bug-triggering interleavings
AVPredictor
Academic
Linux
Parallel bug-finding in concurrent programs via reduced interleaving instances
Linux
HPC
C
Structural testing
No Information
Unit
No Information
Concurrent code
Non-deterministic sequential C program
No Information
Case study
Static
Shared memory
No Information
No Information
No Information
No Information
Bug-triggering interleavings
Lazy-CSeq
Academic
Linux