Sample Submission Dataset
A plagiarism engine is only as trustworthy as the evidence used to test it. If you run the pipeline on files you have not examined, you cannot tell whether a 62% external match means the engine is working or misfiring. The sample_submissions directory solves this problem. It is a small, deliberately constructed corpus in which you know the correct answer for every file before the engine runs. That knowledge is what lets you compute precision, recall, and the pass/fail outcomes in the test scenarios, and it is what your EVALUATION.md must be measured against.
The corpus does three jobs at once. It exercises every ingestion path (PDF, DOCX, MP4) that Module 1 must handle. It supplies the duplicate, cross-modal, and AI-generated cases that the similarity and integrity modules must detect. It also supplies the controls and degraded inputs that test whether the engine avoids false accusations and survives bad files. Each file therefore has a purpose, and the manifest records that purpose. Treat the dataset as a designed experiment rather than a folder of documents gathered at random.
Corpus Composition Requirements
The corpus must contain a minimum mix of file types: at least 5 PDFs, at least 5 DOCX files, and 2 to 3 MP4 clips. Each clip should run 10 to 30 seconds and contain spoken English, because the audio must be transcribed before it can be compared with text. The total should land between 10 and 15 files. The per-type counts are floors, not targets, so a corpus of five PDFs, five DOCX files, and three MP4 clips already sits at the upper end of the total range. Planted cases described below count toward these numbers rather than adding to them.
All content must be synthetic, public, or open-domain. You may write the essays yourself, use public-domain texts, or generate text with a language model and disclose that. You must not use private student work, copyrighted material you do not have rights to, or anything that would make the repository sensitive to share. This rule keeps the corpus reproducible: anyone who clones the repository can rerun the pipeline and obtain the same results. The detailed rules for choosing topics, lengths, and formatting are covered in the child article on corpus composition.
Planted Duplicate, Cross-Modal, LLM, and Original Cases
A planted case is a file whose relationship to another file you control on purpose. The corpus requires four kinds. An exact-copy pair is two files, in any formats, that share an identical passage; this tests whether the engine finds verbatim reuse across documents. A cross-modal pair is an MP4 whose speech closely follows a section of a PDF; this tests whether transcripts are compared against text with page and timestamp references intact. An LLM-generated document is a text written by a language model, which gives the AI-content signal something to detect. Two completely original submissions act as false-positive controls, because a detector that flags everything looks perfect on duplicates alone.
Each planted case maps to a specific expectation in the test scenarios. The exact-copy pair should produce a high external or peer match for the affected documents. The cross-modal pair maps to Scenario 1, where the expected output names the PDF page and the MP4 timestamp. The LLM document should raise the AI content likelihood above that of the human-written files. The originals should score low across all three signals. The planted case table in the child article specifies the exact expected ranges. The figure below summarises how the four case types connect to their expected outcomes.
Degraded and Malformed Test Files
The corpus must include at least one degraded file: a silent MP4, a zero-byte file, or a heavily formatted PDF. Real submission systems receive files like these constantly, and the pipeline's behaviour on them matters as much as its behaviour on clean inputs. The principle is that a bad file should produce a warning and a recorded outcome, never a crash that halts the batch. Scenario 3 requires the engine to warn, skip or flag, and continue.
Each degraded type has a distinct expected behaviour. A zero-byte file contains nothing to parse, so the correct outcome is a skip with a logged warning. A silent MP4 has a valid container but no speech, so transcription returns empty text and the correct outcome is a flag indicating no usable audio. A heavily formatted PDF, with multi-column layouts, headers, and dense tables, still contains text, so the expected outcome is successful extraction after normalization, possibly with a warning about reduced reliability. Including one file of each kind is a strong choice, though the minimum is one. The child article explains how to construct each file and what the log line must say.
dataset_manifest.json Ground Truth Format
The manifest is the machine-readable record of what the corpus contains and what the engine should conclude about it. For each file it records the file's type, its expected labels (for example, whether it is a duplicate, a cross-modal match, AI-generated, original, or degraded), any duplicate pairs it belongs to, and an expected score range for each of the three signals. The manifest is written by you before running the engine, which is what makes it ground truth rather than a summary of the output.
The manifest feeds the evaluation directly. Expected labels, compared with the engine's flags, produce a confusion matrix and from it precision and recall. Duplicate pairs define the set of true matches that recall is measured against, so a missed pair lowers recall and a spurious pair lowers precision. Expected score ranges let the evaluation check whether each score falls where it should, which tests calibration as well as classification. The table below lists the required fields and their role. The child article gives the exact schema and an example entry.
| Manifest field | What it records | How the evaluation uses it |
|---|---|---|
| file_name | Path of the file within the corpus | Joins manifest entries to engine reports |
| file_type | PDF, DOCX, or MP4 | Checks ingestion coverage per format |
| expected_labels | Duplicate, cross-modal, AI-generated, original, or degraded | Builds the confusion matrix |
| duplicate_pairs | Files that share planted content with this file | Defines true matches for recall and precision |
| expected_score_ranges | Low and high bounds for each of the three scores | Checks whether scores fall where they should |