Merge pull request #1964 from ludovic111/perf/ci-speed-selection
🤖 Agent: merged on verify. It changes only which harnesses run, not the game; evidence in the body: all 61 failed parity runs of 10-18..10-25 are still caught, and LLVM coverage on 6 harnesses confirms the selection. Its own parity selects no harness. CI speed is Ludovic's priority tonight.
This commit is contained in:
@@ -49,6 +49,10 @@ A harness can declare, at the top level of its script:
|
||||
- `PARITY_SHARDS = N`: it runs as `NAME#1` to `NAME#N`, in parallel, each told its share in `PARITY_SHARD=i/N` (`test_datafix_parity.py` upgrades every sixth world). `NAME` on the command line or in `--known-failures` means all of them.
|
||||
- `PARITY_RUST_SCOPE = [...]`: the Rust sources its probes can reach, as path prefixes. Cargo's dep-info names every file of every crate a probe links, so without it a probe of `horizon-world` depends on all 350 of its files. The runner keeps only the scope's files if the scope is closed: nothing in it names a module, crate-root re-export or macro outside it (`crate::`, `super::`, other workspace crates; comments and strings ignored), declares a module it doesn't cover, or includes source, and nothing outside it implements anything for a type it defines. Crate roots then count without their declarations of other modules, so adding a module elsewhere doesn't rerun the harness. If the scope isn't closed the runner says why and keeps every file. `datafix` depends on 41 files this way, not 354.
|
||||
|
||||
### Which harnesses a change affects
|
||||
|
||||
`--changed` and `--changed-files` narrow each harness's Rust sources to those its probes can reach (`tools/parity_reach.py`), so a change to one module of `horizon-client` no longer reruns every harness whose probe links the crate. The narrowing works on items, not files: from the probe modules a harness ran (or the binary it built), a free item (fn, type, const, static) is reached when a reached item names it in its own module, through a path or an import of a module that is named, or through a glob import; an inherent impl's methods count one by one once their type is reached and their name is mentioned; a trait impl counts once its trait and its type are reached (std's traits and types always are); a macro counts once invoked. Names match by spelling, so the result only over-approximates. A reached file's parent modules count through their `mod` declarations of it, compared with the merge base; Rust files a change adds take part, a dependency that was deleted counts as changed, code under `#[cfg(test)]` never counts, and a harness whose probe can't be found keeps every source. Parses are cached by content in `.local/parity-runner/reach-cache.json`. `--by-crate` turns the narrowing off. On the last 20 pull requests merged on 2026-10-25, the harnesses picked went from 2255 to 1061 (2384 to 1592 harness-minutes; game-binary harnesses, which link all of `game/`, set the floor), and every harness a CI parity run failed in the week before was still picked; every source file a sample of probes executed (LLVM `-C instrument-coverage`) is among those kept. Building the graph takes 15 s cold, under a second warm, and up to a minute and a half on a change that reaches many harnesses.
|
||||
|
||||
## The gate
|
||||
|
||||
`tools/gate.py` is what a branch must pass before it is integrated: the public-tree check, rustfmt, clippy with warnings denied, the Rust and Python tests, then the harnesses whose inputs changed. It stops at the first failure and writes `.local/gate.json`. `--known-failures a,b` lets harnesses that already fail on the integration branch keep failing. `--quick` checks formatting, lints and Python tests only; `--no-parity` skips the harnesses.
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""tools/parity_reach.py: which workspace sources a harness's probe can reach."""
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
|
||||
from tools import parity, parity_reach
|
||||
|
||||
PROBE = [["example", "world_probe", "release", ["-p", "a"], True]]
|
||||
|
||||
|
||||
def workspace(root, files):
|
||||
for path, text in files.items():
|
||||
(root / path).parent.mkdir(parents=True, exist_ok=True)
|
||||
(root / path).write_text(textwrap.dedent(text), encoding="utf-8")
|
||||
|
||||
|
||||
class ReachTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.directory = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.directory.name)
|
||||
workspace(self.root, {
|
||||
"crates/a/Cargo.toml": '[package]\nname = "a"\n\n[[example]]\nname = "world_probe"\npath = "examples/probe/main.rs"\n',
|
||||
"crates/a/src/lib.rs": """
|
||||
pub mod used;
|
||||
pub mod unused;
|
||||
mod methods;
|
||||
mod display;
|
||||
mod table;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
""",
|
||||
"crates/a/src/used.rs": """
|
||||
pub struct Thing;
|
||||
pub fn helper() -> u32 { 1 }
|
||||
""",
|
||||
# Same names as used.rs, but nothing names this module.
|
||||
"crates/a/src/unused.rs": "pub struct Thing;\npub fn helper() -> u32 { 2 }\n",
|
||||
"crates/a/src/methods.rs": """
|
||||
use crate::used::Thing;
|
||||
impl Thing {
|
||||
pub fn method(&self) -> u32 { 3 }
|
||||
pub fn never_called(&self) -> u32 { 4 }
|
||||
}
|
||||
""",
|
||||
"crates/a/src/display.rs": """
|
||||
impl std::fmt::Display for crate::used::Thing {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { write!(f, "thing") }
|
||||
}
|
||||
""",
|
||||
"crates/a/src/table.rs": """
|
||||
macro_rules! table { () => { pub struct Table; }; }
|
||||
table!();
|
||||
""",
|
||||
"crates/a/src/tests.rs": "use crate::used::Thing;\nimpl Thing { pub fn method_in_tests(&self) {} }\n",
|
||||
"crates/a/examples/probe/main.rs": "mod dispatch;\nmod p;\n",
|
||||
"crates/a/examples/probe/dispatch.rs": "pub fn args() {}\n",
|
||||
"crates/a/examples/probe/p.rs": """
|
||||
use a::used::{helper, Thing};
|
||||
pub fn main() { println!("{} {} {}", helper(), Thing.method(), Thing); }
|
||||
""",
|
||||
})
|
||||
self.dependencies = sorted(str(path.relative_to(self.root)) for path in self.root.rglob("*") if path.is_file())
|
||||
self.dependencies.remove("crates/a/examples/probe/main.rs") # parity.only_probes_run leaves it out
|
||||
|
||||
def tearDown(self):
|
||||
self.directory.cleanup()
|
||||
|
||||
def narrowed(self, changed=(), artifacts=PROBE):
|
||||
graph = parity_reach.SourceGraph(self.root, parity.rust_code)
|
||||
return parity_reach.narrow(self.dependencies, artifacts, graph, set(changed))
|
||||
|
||||
def test_keeps_what_the_probe_reaches(self):
|
||||
kept = {path for path, (whole, _) in self.narrowed().items() if whole}
|
||||
self.assertIn("crates/a/src/used.rs", kept)
|
||||
self.assertIn("crates/a/src/methods.rs", kept) # Thing.method()
|
||||
self.assertIn("crates/a/src/display.rs", kept) # std's Display for a reached type
|
||||
self.assertIn("crates/a/Cargo.toml", kept)
|
||||
self.assertNotIn("crates/a/src/unused.rs", kept) # its names match, but nothing names its module
|
||||
self.assertNotIn("crates/a/src/table.rs", kept)
|
||||
self.assertNotIn("crates/a/src/tests.rs", kept) # compiled into tests only
|
||||
|
||||
def test_module_parents_count_through_their_declarations(self):
|
||||
self.assertEqual(self.narrowed()["crates/a/src/lib.rs"], (False, {"used", "methods", "display"}))
|
||||
text = (self.root / "crates/a/src/lib.rs").read_text()
|
||||
self.assertEqual(parity_reach.declarations(text, {"used"}, False), "pub mod used;")
|
||||
self.assertEqual(parity_reach.declarations(text, {"tests"}, False), "#[cfg(test)]\nmod tests;")
|
||||
# Adding or dropping a module beside the reached ones changes neither view.
|
||||
other = text.replace("mod table;", "mod table2;\nmod more;")
|
||||
self.assertEqual(parity_reach.declarations(other, {"used"}, False), "pub mod used;")
|
||||
self.assertEqual(parity_reach.declarations(other, {"used"}, True), parity_reach.declarations(text, {"used"}, True))
|
||||
self.assertNotEqual(parity_reach.declarations(text + "pub fn f() {}\n", {"used"}, True),
|
||||
parity_reach.declarations(text, {"used"}, True))
|
||||
|
||||
def test_a_new_file_counts_once_it_is_reached(self):
|
||||
(self.root / "crates/a/src/lib.rs").write_text("pub mod used;\nmod extra;\n")
|
||||
workspace(self.root, {"crates/a/src/extra.rs": "impl crate::used::Thing { pub fn method(&self) -> u32 { 5 } }\n"})
|
||||
self.assertIn("crates/a/src/extra.rs", self.narrowed(changed={"crates/a/src/extra.rs"}))
|
||||
|
||||
def test_a_deleted_dependency_counts(self):
|
||||
(self.root / "crates/a/src/unused.rs").unlink()
|
||||
self.assertIn("crates/a/src/unused.rs", self.narrowed())
|
||||
|
||||
def test_unknown_probes_keep_everything(self):
|
||||
self.assertEqual(set(self.narrowed(artifacts=[])), set(self.dependencies))
|
||||
self.assertEqual(set(self.narrowed(artifacts=[["example", "missing", "release", ["-p", "a"], True]])),
|
||||
set(self.dependencies))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+42
-7
@@ -45,6 +45,11 @@ import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
try:
|
||||
from . import parity_reach
|
||||
except ImportError:
|
||||
import parity_reach
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LOCAL = ROOT / ".local"
|
||||
STATE_DIR = LOCAL / "parity-runner"
|
||||
@@ -1224,20 +1229,44 @@ def changed_harnesses(found, records, ref):
|
||||
return found_changed
|
||||
|
||||
|
||||
def affected_harnesses(found, records, changed):
|
||||
def affected_harnesses(found, records, changed, reach=True, base=None):
|
||||
"""Harnesses the changed files can affect: their script or a recorded dependency (here, or in the shared
|
||||
dependency lists) changed, a file every harness depends on changed, or nothing is recorded about them yet."""
|
||||
dependency lists) changed, a file every harness depends on changed, or nothing is recorded about them yet.
|
||||
With reach, a harness's workspace Rust sources count only if its probes can reach them (tools/parity_reach.py);
|
||||
a harness whose probes can't be placed keeps them all. A changed file counts unless, compared with base (a
|
||||
commit), only declarations of modules the harness doesn't reach changed (tools/parity_reach.py)."""
|
||||
if changed & SHARED_INPUTS:
|
||||
return set(found)
|
||||
graph = None
|
||||
if reach and any(path.startswith("crates/") for path in changed):
|
||||
cache_path = STATE_DIR / "reach-cache.json"
|
||||
try:
|
||||
cache = json.loads(cache_path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
cache = {}
|
||||
graph = parity_reach.SourceGraph(ROOT, rust_code, cache)
|
||||
affected = set()
|
||||
for name, script in found.items():
|
||||
dependencies = records.get(name, {}).get("dependencies")
|
||||
record = records.get(name) or {}
|
||||
dependencies = record.get("dependencies")
|
||||
template = SHARED / "dependencies" / f"{name}.json"
|
||||
if dependencies is None and template.exists():
|
||||
dependencies = json.loads(template.read_text(encoding="utf-8")).get("dependencies")
|
||||
if dependencies is None or Path(script).relative_to(ROOT).as_posix() in changed or \
|
||||
any(path.split(KEEP)[0] in changed for path in dependencies):
|
||||
record = json.loads(template.read_text(encoding="utf-8"))
|
||||
dependencies = record.get("dependencies")
|
||||
if dependencies is None or Path(script).relative_to(ROOT).as_posix() in changed:
|
||||
affected.add(name)
|
||||
continue
|
||||
plain = {path.split(KEEP)[0] for path in dependencies}
|
||||
if graph is not None and plain & changed:
|
||||
narrowed = parity_reach.narrow(dependencies, recorded_artifacts(record), graph, changed)
|
||||
if any(path.split(KEEP)[0] in changed and (children is None or parity_reach.declarations_changed(
|
||||
ROOT, path.split(KEEP)[0], children, base, whole)) for path, (whole, children) in narrowed.items()):
|
||||
affected.add(name)
|
||||
elif plain & changed:
|
||||
affected.add(name)
|
||||
if graph is not None:
|
||||
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
cache_path.write_text(json.dumps(graph.saved_cache()), encoding="utf-8")
|
||||
return affected
|
||||
|
||||
|
||||
@@ -1280,6 +1309,8 @@ def main():
|
||||
parser.add_argument("--flake-rounds", type=int, default=3, help="runs required by --flake-check (default 3)")
|
||||
parser.add_argument("--changed", metavar="REF", help="only harnesses the changes since the merge base with REF can "
|
||||
"affect (and that have not passed with their current inputs): the pull-request check")
|
||||
parser.add_argument("--by-crate", action="store_true", help="with --changed: count every Rust source a probe "
|
||||
"links, not only those it can reach (tools/parity_reach.py)")
|
||||
parser.add_argument("--changed-files", metavar="FILE", help="like --changed, for the changed paths listed in FILE, "
|
||||
"one per line: for a copy without Git history (tools/pr_ready.py on another machine)")
|
||||
args = parser.parse_args()
|
||||
@@ -1316,7 +1347,11 @@ def main():
|
||||
changed, since = changed_files(args.changed), f" since the merge base with {args.changed}"
|
||||
if changed is None:
|
||||
parser.error(f"no merge base with {args.changed}")
|
||||
affected = affected_harnesses(found, records, changed)
|
||||
base = None
|
||||
if args.changed:
|
||||
merge_base = subprocess.run(["git", "merge-base", "HEAD", args.changed], cwd=ROOT, capture_output=True, text=True)
|
||||
base = merge_base.stdout.strip() or None
|
||||
affected = affected_harnesses(found, records, changed, reach=not args.by_crate, base=base)
|
||||
chosen = [name for name in found if name in affected and stale(name, state, hasher, environment)]
|
||||
print(f"{len(changed)} files changed{since}: {len(affected)} harnesses "
|
||||
f"depend on them, {len(chosen)} without a pass for their current inputs.", flush=True)
|
||||
|
||||
@@ -0,0 +1,768 @@
|
||||
"""Which of a harness's Rust sources its probes can reach, for `parity.py --changed`.
|
||||
|
||||
Cargo's dep-info names every source file of every crate a probe links, so a change anywhere in horizon-client
|
||||
selected every harness whose probe links horizon-client. This module narrows a harness's recorded dependencies to
|
||||
the files holding an item its entry points (the probe modules it ran, the binaries it built) can reach, over a graph
|
||||
of the workspace's current sources whose nodes are items:
|
||||
|
||||
- everything in an entry is reached, and so is everything in a Rust file the module tree doesn't place (an
|
||||
`include!`d source);
|
||||
- a module is open when it is a crate root, holds reached code, or is named in a path (`name::`, `use ...name`) by
|
||||
reached code while its parent is open;
|
||||
- a free item (fn, struct, enum, union, trait, type, const, static, an item-level macro call's items) is reached when
|
||||
reached code in its own module or below it mentions its name, or when its module is open and reached code names
|
||||
it in a path or an import, or when its module is glob-imported (`use m::*`) and its name is mentioned;
|
||||
- an inherent impl's methods and constants are reached one by one, by name, once their type is; a trait impl once
|
||||
its trait and its type are (a std or outside name always is, a generic parameter needs nothing); a macro once
|
||||
invoked; a `pub use` like a free item named by everything it imports;
|
||||
- "a crate's reached code" covers the crates it names (`horizon_world::`), directly or through one that does, and
|
||||
code under `#[cfg(test)]` or `#[test]` is left out (probes are never test builds).
|
||||
|
||||
Names match by spelling, so a file can be kept for a name it never meant: the graph only over-approximates. A
|
||||
reached file's parent modules count through their `mod` declarations of it (parity.py compares those with the merge
|
||||
base). Everything that isn't a workspace Rust source stays a dependency, except a crate's non-Rust files (shaders,
|
||||
assets, fonts) that only unreached sources name; a dependency that no longer exists counts as changed, and Rust files
|
||||
a change adds take part.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
IDENT = r"[A-Za-z_]\w*"
|
||||
WORD = re.compile(rf"\b({IDENT})\b")
|
||||
MOD_DECLARATION = re.compile(rf"\bmod\s+({IDENT})\s*;")
|
||||
INLINE_MOD = re.compile(rf"\bmod\s+({IDENT})\s*\{{")
|
||||
PATH_ATTRIBUTE = re.compile(r'#\[\s*path\s*=\s*"([^"]+)"\s*\]\s*(?:#\[[^\]]*\]\s*)*(?:pub(?:\([^)]*\))?\s+)?mod\s+\w+\s*;$')
|
||||
BRACE = re.compile(r"[{}]")
|
||||
TEST_ONLY = re.compile(r"#\[\s*(?:cfg\s*\(\s*test\s*\)|test)\s*\]")
|
||||
TEST_MODULE = re.compile(r"#\[\s*cfg\s*\(\s*test\s*\)\s*\]\s*(?:#\[[^\]]*\]\s*)*(?:pub(?:\([^)]*\))?\s+)?mod\s+\w+\s*;$")
|
||||
SPLIT = re.compile(r"[{}()\[\];]")
|
||||
ITEM = re.compile(rf"(?:pub(?:\s*\([^)]*\))?\s+)?(?:(?:unsafe|async|const|extern|default|auto)\s+)*"
|
||||
rf"(fn|struct|enum|union|trait|type|const|static|impl|mod|use|macro_rules\s*!|extern\s+crate)"
|
||||
rf"(?!\w)\s*(?:mut\s+)?({IDENT})?")
|
||||
PATH_BEFORE = re.compile(rf"\b({IDENT})\s*::")
|
||||
PATH_AFTER = re.compile(rf"::\s*({IDENT})")
|
||||
MACRO_USE = re.compile(rf"\b({IDENT})\s*!\s*[(\[{{]")
|
||||
MACRO_CALL = re.compile(rf"({IDENT}(?:\s*::\s*{IDENT})*)\s*!")
|
||||
DEFINED_INSIDE = re.compile(rf"\b(?:fn|struct|enum|union|trait|type|const|static)\s+({IDENT})")
|
||||
NESTED_IMPL = re.compile(r"^[ \t]*(?:unsafe\s+)?impl\b([^{;]*)\{", re.MULTILINE)
|
||||
QUALIFIED = re.compile(rf"{IDENT}(?:\s*::\s*{IDENT})*")
|
||||
GLOB_IMPORT = re.compile(r"\buse\s[^;]*::\s*\*")
|
||||
# File names a source may name in a literal (include_str!, include_bytes!, a path it reads). A file of another kind
|
||||
# is never "named", so it always counts.
|
||||
FILE_NAME = re.compile(r"[\w-]+\.(?:wgsl|glsl|json|png|ttf|otf|svg|txt|toml|ogg|wav|nbt|bin|dat|csv|mcmeta|wit|wasm)\b")
|
||||
TYPE_KINDS = {"struct", "enum", "union", "trait", "type", "macro_call"}
|
||||
# Names an impl header can take from outside the workspace: std's prelude and primitive types, and the std and serde
|
||||
# traits a glob import may bring. A glob-imported name outside this list is taken to be the workspace's.
|
||||
STD_NAMES = {"Display", "Error", "Read", "Write", "Seek", "BufRead", "Deref", "DerefMut", "Index", "IndexMut", "Add",
|
||||
"Sub", "Mul", "Div", "Rem", "Neg", "Not", "BitAnd", "BitOr", "BitXor", "Shl", "Shr", "AddAssign",
|
||||
"SubAssign", "MulAssign", "DivAssign", "RemAssign", "BitAndAssign", "BitOrAssign", "BitXorAssign",
|
||||
"Borrow", "BorrowMut", "Future", "Any", "FromStr", "Hasher", "BuildHasher", "Formatter", "Ordering",
|
||||
"Serialize", "Deserialize", "Serializer", "Deserializer", "Visitor", "Termination", "Sum", "Product",
|
||||
"Wake", "Cow", "Rc", "Arc", "Mutex", "RwLock", "Cell", "RefCell", "HashMap", "HashSet", "BTreeMap",
|
||||
"BTreeSet", "VecDeque"}
|
||||
PRELUDE = {"Option", "Some", "None", "Result", "Ok", "Err", "Vec", "String", "Box", "ToString", "ToOwned", "Clone",
|
||||
"Copy", "Send", "Sync", "Sized", "Unpin", "Drop", "Fn", "FnMut", "FnOnce", "From", "Into", "TryFrom",
|
||||
"TryInto", "AsRef", "AsMut", "Default", "Eq", "PartialEq", "Ord", "PartialOrd", "Iterator", "Extend",
|
||||
"IntoIterator", "DoubleEndedIterator", "ExactSizeIterator", "FromIterator", "Debug", "Hash",
|
||||
"bool", "char", "str", "u8", "u16", "u32", "u64", "u128", "usize", "i8", "i16", "i32", "i64", "i128",
|
||||
"isize", "f32", "f64"}
|
||||
STD_NAMES |= PRELUDE
|
||||
KEYWORDS = {"crate", "self", "super", "Self", "pub", "in", "as", "use", "for", "where", "dyn", "mut", "const", "fn",
|
||||
"unsafe", "impl", "std", "core", "alloc", "let", "if", "else", "match", "return", "struct", "enum", "mod",
|
||||
"type", "static", "trait", "true", "false", "loop", "while", "move", "ref", "break", "continue", "async",
|
||||
"await", "extern", "union"}
|
||||
|
||||
|
||||
def segments(code, start, end):
|
||||
"""The item-level pieces of code[start:end]: each ends at a `;` or a `}` outside any bracket."""
|
||||
found, begin, depth = [], start, 0
|
||||
for match in SPLIT.finditer(code, start, end):
|
||||
character = match.group()
|
||||
if character in "{([":
|
||||
depth += 1
|
||||
elif character in "})]":
|
||||
depth -= 1
|
||||
if character == "}" and depth == 0:
|
||||
found.append((begin, match.end()))
|
||||
begin = match.end()
|
||||
elif depth == 0:
|
||||
found.append((begin, match.end()))
|
||||
begin = match.end()
|
||||
if code[begin:end].strip():
|
||||
found.append((begin, end))
|
||||
return found
|
||||
|
||||
|
||||
def skip_attributes(code, index, end):
|
||||
"""The offset of the first non-space character after the attributes (`#[...]`, `#![...]`) starting at index."""
|
||||
while True:
|
||||
while index < end and code[index].isspace():
|
||||
index += 1
|
||||
if not code.startswith("#", index) or not re.match(r"#!?\[", code[index:index + 3]):
|
||||
return index
|
||||
depth = 0
|
||||
for position in range(code.index("[", index), end):
|
||||
depth += {"[": 1, "]": -1}.get(code[position], 0)
|
||||
if depth == 0:
|
||||
index = position + 1
|
||||
break
|
||||
else:
|
||||
return end
|
||||
|
||||
|
||||
def body_of(code, start, end):
|
||||
"""The range inside an item's outermost `{ }` block, or None."""
|
||||
opening = code.find("{", start, end)
|
||||
closing = code.rfind("}", start, end)
|
||||
return (opening + 1, closing) if 0 <= opening < closing else None
|
||||
|
||||
|
||||
def words(text):
|
||||
return set(WORD.findall(text)) - KEYWORDS
|
||||
|
||||
|
||||
def qualified_words(text):
|
||||
"""Names a piece of code reaches through a path (`a::b`: b), the way items of other modules are named."""
|
||||
return set(PATH_AFTER.findall(text)) - KEYWORDS
|
||||
|
||||
|
||||
def path_words(text):
|
||||
"""Names a piece of code uses as path segments (`a::`, `::b`), where module names appear."""
|
||||
return set(PATH_BEFORE.findall(text)) | set(PATH_AFTER.findall(text))
|
||||
|
||||
|
||||
def parse_items(code, start, end, items):
|
||||
"""Append the items of code[start:end] (a file or an inline module's body) to items, as dicts."""
|
||||
for begin, finish in segments(code, start, end):
|
||||
head = skip_attributes(code, begin, finish)
|
||||
text = code[head:finish]
|
||||
if not text.strip(" \n\t;") or TEST_ONLY.search(code[begin:head]):
|
||||
continue # compiled into tests only, never into a probe
|
||||
match = ITEM.match(text)
|
||||
kind = re.sub(r"\s+", "", match.group(1)) if match else None
|
||||
name = match.group(2) if match else None
|
||||
if kind == "mod":
|
||||
body = body_of(code, head, finish)
|
||||
if body:
|
||||
parse_items(code, body[0], body[1], items)
|
||||
continue
|
||||
if kind in ("externcrate", None) and not MACRO_CALL.match(text):
|
||||
if kind is None:
|
||||
items.append({"kind": "glue", "mentions": sorted(words(text)), "paths": sorted(path_words(text)), "qualified": sorted(qualified_words(text)), "invokes": sorted(set(MACRO_USE.findall(text)))})
|
||||
continue
|
||||
if kind is None: # an item-level macro call: it may define items
|
||||
items.append({"kind": "macro_call", "names": sorted(set(DEFINED_INSIDE.findall(text))),
|
||||
"macro": MACRO_CALL.match(text).group(1).split("::")[-1].strip(),
|
||||
"mentions": sorted(words(text)), "paths": sorted(path_words(text)), "qualified": sorted(qualified_words(text)), "invokes": sorted(set(MACRO_USE.findall(text)))})
|
||||
elif kind == "use":
|
||||
head_word = re.match(rf"\s*(?:::\s*)?({IDENT})", text[match.end(1):])
|
||||
items.append({"kind": "use", "public": text.startswith("pub"), "names": sorted(words(text)),
|
||||
"head": head_word.group(1) if head_word else "", "glob": bool(re.search(r"::\s*\*", text)),
|
||||
"mentions": sorted(words(text)), "paths": sorted(words(text)), "qualified": sorted(words(text)),
|
||||
"invokes": []})
|
||||
elif kind == "impl":
|
||||
body = body_of(code, head, finish)
|
||||
subject = code[head + match.end(1) - match.start(): body[0] - 1 if body else finish]
|
||||
inherent = not re.search(r"\bfor\b", impl_subject(subject)[1])
|
||||
if inherent and body:
|
||||
members = []
|
||||
for member_begin, member_end in segments(code, body[0], body[1]):
|
||||
member = code[skip_attributes(code, member_begin, member_end):member_end]
|
||||
found = ITEM.match(member)
|
||||
members.append({"name": found.group(2) if found else None, "mentions": sorted(words(member)),
|
||||
"paths": sorted(path_words(member)), "qualified": sorted(qualified_words(member)), "invokes": sorted(set(MACRO_USE.findall(member)))})
|
||||
items.append({"kind": "impl", "subject": subject, "inherent": True, "members": members,
|
||||
"mentions": sorted(words(subject)), "paths": sorted(path_words(subject)), "qualified": sorted(qualified_words(subject)), "invokes": sorted(set(MACRO_USE.findall(subject)))})
|
||||
else:
|
||||
items.append({"kind": "impl", "subject": subject, "inherent": False, "mentions": sorted(words(text)),
|
||||
"paths": sorted(path_words(text)), "qualified": sorted(qualified_words(text)), "invokes": sorted(set(MACRO_USE.findall(text)))})
|
||||
else:
|
||||
kind = "macro" if kind.startswith("macro_rules") else kind
|
||||
items.append({"kind": kind, "names": [name] if name else [], "mentions": sorted(words(text)),
|
||||
"defines": sorted(set(DEFINED_INSIDE.findall(text))) if kind == "macro" else [],
|
||||
"paths": sorted(path_words(text)), "qualified": sorted(qualified_words(text)), "invokes": sorted(set(MACRO_USE.findall(text)))})
|
||||
if kind in ("fn", "const", "static"):
|
||||
# An impl inside a function body still holds for the whole program.
|
||||
for header in NESTED_IMPL.findall(text):
|
||||
items.append({"kind": "impl", "subject": header, "inherent": False,
|
||||
"mentions": sorted(words(text)), "paths": sorted(path_words(text)), "qualified": sorted(qualified_words(text)), "invokes": sorted(set(MACRO_USE.findall(text)))})
|
||||
|
||||
|
||||
def crate_roots(crate_dir):
|
||||
"""A crate's root files: its library, binaries and examples."""
|
||||
roots = [crate_dir / "src" / "lib.rs", crate_dir / "src" / "main.rs"]
|
||||
for folder in (crate_dir / "src" / "bin", crate_dir / "examples"):
|
||||
if folder.is_dir():
|
||||
roots += sorted(folder.glob("*.rs")) + sorted(folder.glob("*/main.rs"))
|
||||
return [root for root in roots if root.is_file()]
|
||||
|
||||
|
||||
class SourceGraph:
|
||||
"""The workspace's Rust sources, parsed once: what each file names, defines and declares."""
|
||||
|
||||
def __init__(self, root, rust_code, cache=None):
|
||||
"""cache: a dict of earlier parses by content digest, which the graph reuses and adds to (parity.py keeps it
|
||||
in .local/parity-runner/reach-cache.json)."""
|
||||
self.root = Path(root)
|
||||
self.rust_code = rust_code
|
||||
self.cache = {} if cache is None else cache
|
||||
self.used = set()
|
||||
self.files = {}
|
||||
self.parent = {} # module file -> the file declaring it
|
||||
self.module_name = {}
|
||||
self.roots = set()
|
||||
self.target_cache = {}
|
||||
self.workspace_heads = None
|
||||
self.test_only = set() # modules compiled into tests only
|
||||
self.tables = None
|
||||
self.root_of = {} # source -> the root of its target (lib.rs, a bin's or an example's main file)
|
||||
self.results = {} # (entries, universe) -> reachable files
|
||||
for manifest in sorted((self.root / "crates").glob("*/Cargo.toml")):
|
||||
for crate_root in crate_roots(manifest.parent):
|
||||
path = self.relative(crate_root)
|
||||
self.roots.add(path)
|
||||
self.root_of[path] = path
|
||||
self.place(path, crate_root.parent, set())
|
||||
|
||||
def entries(self, artifacts, rust):
|
||||
"""The root files of the probes and binaries a harness built (its record's artifacts), among its Rust
|
||||
dependencies: for a crate's probe binary, the probe modules it ran (parity.only_probes_run kept only those).
|
||||
None when an artifact can't be placed."""
|
||||
found = set()
|
||||
for kind, name, *rest in artifacts:
|
||||
selection = rest[1] if len(rest) > 1 else []
|
||||
packages = [value for flag, value in zip(selection, selection[1:]) if flag == "-p"] or \
|
||||
[manifest.parent.name for manifest in sorted((self.root / "crates").glob("*/Cargo.toml"))]
|
||||
roots = set()
|
||||
for package in packages:
|
||||
crate = self.root / "crates" / package
|
||||
targets = self.targets(crate)
|
||||
if kind in ("example", "examples"):
|
||||
candidates = [path for target, (target_kind, path) in targets.items()
|
||||
if target_kind == "example" and (not name or target == name)]
|
||||
else:
|
||||
candidates = [path for target, (target_kind, path) in targets.items()
|
||||
if target_kind == "bin" and (not name or target == name)]
|
||||
for path in candidates:
|
||||
if path.endswith("/examples/probe/main.rs"):
|
||||
# The probes it ran, and the dispatch and support modules they use (only_probes_run kept
|
||||
# those); the binary's root, which only declares them, so their own submodules are placed.
|
||||
folder = path.removesuffix("main.rs")
|
||||
roots |= {dependency for dependency in rust if dependency.startswith(folder)
|
||||
and "/" not in dependency.removeprefix(folder).removeprefix("support/")}
|
||||
if roots:
|
||||
roots.add(path)
|
||||
elif path in rust:
|
||||
roots.add(path)
|
||||
if not roots:
|
||||
return None
|
||||
found |= roots
|
||||
return found or None
|
||||
|
||||
def targets(self, crate):
|
||||
"""A crate's bins and examples: {target name: (kind, root file)}, from its Cargo.toml and Cargo's defaults."""
|
||||
if crate in self.target_cache:
|
||||
return self.target_cache[crate]
|
||||
found = {}
|
||||
manifest = (crate / "Cargo.toml").read_text(encoding="utf-8") if (crate / "Cargo.toml").is_file() else ""
|
||||
package = re.search(r'^\[package\][^\[]*?^name\s*=\s*"([^"]+)"', manifest, re.MULTILINE | re.DOTALL)
|
||||
if (crate / "src" / "main.rs").is_file() and package:
|
||||
found[package.group(1)] = ("bin", self.relative(crate / "src" / "main.rs"))
|
||||
for kind, folder in (("bin", crate / "src" / "bin"), ("example", crate / "examples")):
|
||||
for path in sorted(folder.glob("*.rs")) + sorted(folder.glob("*/main.rs")):
|
||||
found[path.stem if path.name != "main.rs" else path.parent.name] = (kind, self.relative(path))
|
||||
for kind, table in re.findall(r"^\[\[(bin|example)\]\]([^\[]*)", manifest, re.MULTILINE):
|
||||
name = re.search(r'^name\s*=\s*"([^"]+)"', table, re.MULTILINE)
|
||||
path = re.search(r'^path\s*=\s*"([^"]+)"', table, re.MULTILINE)
|
||||
if name and path and (crate / path.group(1)).is_file():
|
||||
found[name.group(1)] = (kind, self.relative(crate / path.group(1)))
|
||||
self.target_cache[crate] = found
|
||||
return found
|
||||
|
||||
def target(self, path):
|
||||
"""The Cargo target a source belongs to: the root file of the library, bin or example whose module tree
|
||||
holds it (a file nothing places is its own)."""
|
||||
return self.root_of.get(path, path)
|
||||
|
||||
def is_library(self, path):
|
||||
return self.target(path).endswith("/src/lib.rs")
|
||||
|
||||
def relative(self, path):
|
||||
return Path(path).resolve().relative_to(self.root.resolve()).as_posix()
|
||||
|
||||
def place(self, path, module_dir, seen):
|
||||
"""Walk the module tree from a file: record each declared module's file, parent and name."""
|
||||
if path in seen:
|
||||
return
|
||||
seen.add(path)
|
||||
parsed = self.parse(path)
|
||||
if parsed is None:
|
||||
return
|
||||
for name, nesting, attribute, test in parsed["declarations"]:
|
||||
if attribute is not None:
|
||||
if nesting:
|
||||
continue # a #[path] inside an inline module: unplaced, so always kept
|
||||
candidates = [(self.root / path).parent / attribute]
|
||||
else:
|
||||
folder = module_dir.joinpath(*nesting)
|
||||
candidates = [folder / f"{name}.rs", folder / name / "mod.rs"]
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
child = self.relative(candidate)
|
||||
self.parent.setdefault(child, path)
|
||||
self.module_name.setdefault(child, name)
|
||||
self.root_of.setdefault(child, self.root_of[path])
|
||||
if test or path in self.test_only:
|
||||
self.test_only.add(child)
|
||||
child_dir = candidate.parent if candidate.name == "mod.rs" or attribute is not None \
|
||||
else candidate.parent / candidate.stem
|
||||
self.place(child, child_dir, seen)
|
||||
break
|
||||
|
||||
def parse(self, path):
|
||||
if path in self.files:
|
||||
return self.files[path]
|
||||
try:
|
||||
data = (self.root / path).read_bytes()
|
||||
except OSError:
|
||||
self.files[path] = None
|
||||
return None
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
self.used.add(digest)
|
||||
if digest in self.cache:
|
||||
self.files[path] = parsed = self.cache[digest]
|
||||
return parsed
|
||||
raw = data.decode("utf-8", errors="replace")
|
||||
code = self.rust_code(raw) # same length as raw: literals and comments blanked
|
||||
opening = {match.end() - 1: match.group(1) for match in INLINE_MOD.finditer(code)}
|
||||
events = sorted([(match.start(), match) for match in BRACE.finditer(code)]
|
||||
+ [(match.start(), match) for match in MOD_DECLARATION.finditer(code)], key=lambda event: event[0])
|
||||
declarations, stack, depth = [], [], 0
|
||||
for index, match in events:
|
||||
if match.group() == "{":
|
||||
depth += 1
|
||||
if index in opening:
|
||||
stack.append((opening[index], depth))
|
||||
elif match.group() == "}":
|
||||
if stack and stack[-1][1] == depth:
|
||||
stack.pop()
|
||||
depth -= 1
|
||||
else:
|
||||
window = " ".join(raw[max(0, index - 300):match.end()].split())
|
||||
attribute = PATH_ATTRIBUTE.search(window)
|
||||
declarations.append([match.group(1), [name for name, _ in stack], attribute.group(1) if attribute else None,
|
||||
bool(TEST_MODULE.search(window))])
|
||||
items = []
|
||||
parse_items(code, 0, len(code), items)
|
||||
self.files[path] = parsed = {
|
||||
"declarations": declarations,
|
||||
"items": items,
|
||||
"glob": bool(GLOB_IMPORT.search(code)),
|
||||
"file_names": sorted(set(FILE_NAME.findall(raw))),
|
||||
}
|
||||
self.cache[digest] = parsed
|
||||
return parsed
|
||||
|
||||
def saved_cache(self):
|
||||
"""The parses of the files this graph read, to keep for the next run."""
|
||||
return {digest: self.cache[digest] for digest in self.used if digest in self.cache}
|
||||
|
||||
def requirement(self, path, item):
|
||||
"""For an impl: for its trait and for its type, [names the file defines, other workspace names], one of
|
||||
which must be a reached type or trait (of this file, of a crate it sees) before the impl can matter; both
|
||||
empty: nothing of the workspace is needed. A name is the workspace's when the path naming it starts in the
|
||||
workspace, or the file defines it, imports it from the workspace or may glob-import it; otherwise (std's,
|
||||
another crate's, the prelude's) it needs nothing."""
|
||||
if "requires" in item:
|
||||
return item["requires"]
|
||||
if self.workspace_heads is None:
|
||||
self.workspace_heads = {"crate", "self", "super"} | set(self.module_name.values()) | {
|
||||
manifest.parent.name.replace("-", "_") for manifest in (self.root / "crates").glob("*/Cargo.toml")}
|
||||
parsed = self.parse(path)
|
||||
defined = {name for other in parsed["items"] for name in other.get("names", ()) if other["kind"] != "use"}
|
||||
workspace_imports = {name for other in parsed["items"] if other["kind"] == "use"
|
||||
and other["head"] in self.workspace_heads for name in other["names"]}
|
||||
parameters, subject = impl_subject(item["subject"])
|
||||
parts = []
|
||||
for part in re.split(r"\bfor\b", subject, maxsplit=1):
|
||||
names, own = set(), set()
|
||||
for match in QUALIFIED.finditer(part):
|
||||
segments_ = [segment.strip() for segment in match.group().split("::")]
|
||||
name = segments_[-1]
|
||||
if name in parameters or name in KEYWORDS:
|
||||
continue
|
||||
if len(segments_) > 1:
|
||||
if segments_[0] in self.workspace_heads:
|
||||
names.add(name)
|
||||
elif name in defined:
|
||||
own.add(name)
|
||||
elif name in workspace_imports or (parsed["glob"] and name not in STD_NAMES):
|
||||
names.add(name)
|
||||
parts.append([sorted(own), sorted(names)])
|
||||
item["requires"] = parts
|
||||
return parts
|
||||
|
||||
def index(self):
|
||||
"""Lookup tables over every placed source, built once: which crates each crate sees, the nodes (items, an
|
||||
inherent impl's members) by the names they define and the types they need, the modules by name."""
|
||||
if self.tables is not None:
|
||||
return self.tables
|
||||
idents = {manifest.parent.name.replace("-", "_"): manifest.parent.name
|
||||
for manifest in (self.root / "crates").glob("*/Cargo.toml")}
|
||||
paths = [path for path in self.files if self.files[path] is not None and path not in self.test_only]
|
||||
sees = {}
|
||||
for path in paths:
|
||||
named = {word for item in self.files[path]["items"] for word in item["mentions"]} & idents.keys()
|
||||
sees.setdefault(crate_of(path), {crate_of(path)}).update(idents[name] for name in named)
|
||||
changed = True
|
||||
while changed:
|
||||
changed = False
|
||||
for crate, seen in sees.items():
|
||||
wider = set().union(*(sees.get(other, {other}) for other in seen))
|
||||
if not wider <= seen:
|
||||
seen |= wider
|
||||
changed = True
|
||||
seen_by = {} # crate -> the crates that see it
|
||||
for crate, seen in sees.items():
|
||||
for other in seen:
|
||||
seen_by.setdefault(other, set()).add(crate)
|
||||
# What a call of a workspace macro defines: what its definition's body defines.
|
||||
expands = {}
|
||||
for path in paths:
|
||||
for item in self.files[path]["items"]:
|
||||
if item["kind"] == "macro":
|
||||
for name in item["names"]:
|
||||
expands.setdefault(name, set()).update(item["defines"])
|
||||
names_of, by_name, by_file_name, by_type, by_module, children, seeds = {}, {}, {}, {}, {}, {}, []
|
||||
for path in paths:
|
||||
crate = crate_of(path)
|
||||
if path in self.module_name:
|
||||
by_module.setdefault((crate, self.module_name[path]), []).append(path)
|
||||
children.setdefault(self.parent[path], []).append(path)
|
||||
for index, item in enumerate(self.files[path]["items"]):
|
||||
node = (path, index, None)
|
||||
names = set(item.get("names", ()))
|
||||
if item["kind"] == "macro_call":
|
||||
names |= expands.get(item["macro"], set())
|
||||
names_of[node] = names
|
||||
if item["kind"] == "glue":
|
||||
seeds.append(node)
|
||||
elif item["kind"] == "macro_call" and not names:
|
||||
# A macro call defining nothing by name may still expand to impls: it counts once a type or
|
||||
# trait it mentions is reached.
|
||||
for name in item["mentions"]:
|
||||
by_type.setdefault(name, []).append(node)
|
||||
elif item["kind"] == "impl":
|
||||
required = {name for own, foreign in self.requirement(path, item) for name in own + foreign}
|
||||
members = [(path, index, member) for member in range(len(item["members"]))] if item["inherent"] else []
|
||||
for member_node in members:
|
||||
name = item["members"][member_node[2]]["name"]
|
||||
names_of[member_node] = set()
|
||||
if name:
|
||||
by_name.setdefault(name, []).append(member_node)
|
||||
for name in required:
|
||||
for other in [node] + members:
|
||||
by_type.setdefault(name, []).append(other)
|
||||
if not required:
|
||||
seeds.append(node)
|
||||
seeds += [member_node for member_node in members if not item["members"][member_node[2]]["name"]]
|
||||
else:
|
||||
for name in names:
|
||||
by_name.setdefault(name, []).append(node)
|
||||
by_file_name.setdefault((path, name), []).append(node)
|
||||
self.tables = dict(idents=idents, sees=sees, seen_by=seen_by, names_of=names_of, by_name=by_name,
|
||||
by_file_name=by_file_name, by_type=by_type, by_module=by_module, children=children,
|
||||
seeds=seeds, node_sets={})
|
||||
return self.tables
|
||||
|
||||
def reachable(self, entries, universe, reasons=None):
|
||||
"""The files of universe (workspace Rust sources) holding an item reachable from entries (by the rules in
|
||||
this module's doc), and the entries. reasons, a dict, gets why each file was reached (for debugging)."""
|
||||
# Searching every source of the crates involved (not only those Cargo listed) only adds reach, and lets
|
||||
# harnesses with the same entries (the game binary's) share one search.
|
||||
crates = frozenset(crate_of(path) for path in universe)
|
||||
key = (frozenset(entries), crates)
|
||||
if reasons is None and key in self.results:
|
||||
return self.results[key] & set(universe)
|
||||
everything = {path for path in self.files if crate_of(path) in crates} | set(universe)
|
||||
files = self.search(entries, everything, reasons)
|
||||
self.results[key] = files
|
||||
return files & set(universe)
|
||||
|
||||
def search(self, entries, universe, reasons):
|
||||
targets = {self.target(path) for path in entries}
|
||||
universe = {path for path in universe if (self.is_library(path) or self.target(path) in targets)
|
||||
and path not in self.test_only}
|
||||
tables = self.index()
|
||||
idents, sees, seen_by, names_of = tables["idents"], tables["sees"], tables["seen_by"], tables["names_of"]
|
||||
node_sets = tables["node_sets"]
|
||||
by_name, by_file_name, by_type = tables["by_name"], tables["by_file_name"], tables["by_type"]
|
||||
by_module, children = tables["by_module"], tables["children"]
|
||||
seeds = [node for node in tables["seeds"] if node[0] in universe]
|
||||
for path in universe:
|
||||
if (path in entries or (path not in self.parent and path not in self.roots)) and self.parse(path):
|
||||
seeds += [(path, index, None) for index in range(len(self.parse(path)["items"]))]
|
||||
|
||||
def names(node):
|
||||
return names_of.get(node, set())
|
||||
|
||||
def sets_of(node, item):
|
||||
"""A node's mentions, path words, qualified names and macro calls, as sets (parses keep lists)."""
|
||||
if node not in node_sets:
|
||||
source = item["members"][node[2]] if node[2] is not None else item
|
||||
node_sets[node] = {key: set(source.get(key, ())) for key in ("mentions", "paths", "qualified", "invokes")}
|
||||
return node_sets[node]
|
||||
|
||||
def item_of(node):
|
||||
return self.parse(node[0])["items"][node[1]]
|
||||
|
||||
reached, opened, globbed, files, queue = set(), set(), set(), set(), []
|
||||
# name -> the crates whose reached code mentions it plainly, names it in a path or an import, invokes it.
|
||||
mentioned, qualified, invoked = {}, {}, {}
|
||||
done = {"mentions": {}, "qualified": {}, "invokes": {}, "paths": {}} # crate -> names already handled
|
||||
path_names, types = {}, {} # by crate
|
||||
local = {} # file -> plain mentions by reached code in it or below it (child modules may `use super::*`)
|
||||
local_types = {} # file -> its reached types and traits
|
||||
|
||||
def heard(table, name, crate):
|
||||
"""Whether a crate that can see crate's items has name in table."""
|
||||
sources = table.get(name)
|
||||
return bool(sources) and not sources.isdisjoint(seen_by.get(crate, {crate}))
|
||||
|
||||
def ready(node):
|
||||
path, index, member = node
|
||||
item = item_of(node)
|
||||
crate = crate_of(path)
|
||||
if item["kind"] == "impl":
|
||||
visible = visible_types(crate)
|
||||
here = local_types.get(path, set())
|
||||
if not all((not own and not foreign) or here.intersection(own) or visible.intersection(foreign)
|
||||
for own, foreign in self.requirement(path, item)):
|
||||
return False
|
||||
name = item["members"][member]["name"] if member is not None else None
|
||||
return name is None or heard(mentioned, name, crate) # a method: called by its name
|
||||
if item["kind"] == "macro_call" and not names(node):
|
||||
return not visible_types(crate).isdisjoint(item["mentions"])
|
||||
names_ = names(node)
|
||||
if item["kind"] == "macro":
|
||||
return any(heard(invoked, name, crate) for name in names_)
|
||||
if item["kind"] == "use" and item["glob"]:
|
||||
return path in opened
|
||||
nearby = local.get(path, ())
|
||||
if any(name in nearby for name in names_):
|
||||
return True # named in its own module, or below it
|
||||
if item["kind"] == "use" and not item["public"]:
|
||||
return False
|
||||
if path in opened and any(heard(qualified, name, crate) for name in names_):
|
||||
return True # named through a path, or imported
|
||||
return path in globbed and any(heard(mentioned, name, crate) for name in names_)
|
||||
|
||||
visible_cache = {}
|
||||
|
||||
def visible_types(crate):
|
||||
"""The reached types and traits of the crates a crate sees (cached until a type is reached)."""
|
||||
if crate not in visible_cache:
|
||||
visible_cache[crate] = set().union(*(types.get(other, set()) for other in sees.get(crate, {crate})))
|
||||
return visible_cache[crate]
|
||||
|
||||
def wake(nodes, why):
|
||||
for other in nodes:
|
||||
if other not in reached and other[0] in universe and ready(other):
|
||||
reach(other, why)
|
||||
|
||||
def open_module(path, why):
|
||||
pending = [path]
|
||||
while pending:
|
||||
current = pending.pop()
|
||||
if current in opened:
|
||||
continue
|
||||
opened.add(current)
|
||||
crate = crate_of(current)
|
||||
pending += [child for child in children.get(current, ())
|
||||
if child in universe and self.module_name[child] in path_names.get(crate, ())]
|
||||
parsed = self.parse(current)
|
||||
wake([(current, index, None) for index, item in enumerate(parsed["items"] if parsed else ())
|
||||
if item["kind"] != "impl"], f"{current} opened ({why})")
|
||||
|
||||
def reach(node, why):
|
||||
if node in reached:
|
||||
return
|
||||
reached.add(node)
|
||||
queue.append(node)
|
||||
path, _, member = node
|
||||
item = item_of(node)
|
||||
if (item["kind"] == "use" and not item["public"]) or (item["kind"] == "impl" and item["inherent"] and member is None):
|
||||
return
|
||||
if path not in files:
|
||||
files.add(path)
|
||||
if reasons is not None:
|
||||
reasons[path] = why
|
||||
|
||||
for path in universe:
|
||||
if path.endswith("/src/lib.rs") or path in entries or path not in self.parent:
|
||||
open_module(path, "crate root or entry")
|
||||
for node in seeds:
|
||||
reach(node, "entry" if node[0] in entries else "always counted")
|
||||
for path in entries & universe:
|
||||
files.add(path)
|
||||
if reasons is not None:
|
||||
reasons.setdefault(path, "entry")
|
||||
while queue:
|
||||
node = queue.pop()
|
||||
path, index, member = node
|
||||
item = item_of(node)
|
||||
crate = crate_of(path)
|
||||
label = (item.get("names") or [item["kind"]])[0] if member is None else item["members"][member]["name"]
|
||||
source = sets_of(node, item)
|
||||
why = f"{path} ({label})"
|
||||
# Plain mentions reach items of this file and of the modules above it.
|
||||
ancestor, new = path, source["mentions"]
|
||||
while ancestor is not None and new:
|
||||
known = local.setdefault(ancestor, set())
|
||||
new = new - known
|
||||
known |= new
|
||||
for name in new:
|
||||
wake(by_file_name.get((ancestor, name), ()), f"{why} mentions {name}")
|
||||
ancestor = self.parent.get(ancestor)
|
||||
if item["kind"] == "use" and item["glob"]:
|
||||
for name in item["names"]:
|
||||
for target in sees.get(crate, {crate}):
|
||||
for module in list(by_module.get((target, name), ())) + \
|
||||
([f"crates/{idents[name]}/src/lib.rs"] if name in idents else []):
|
||||
if module not in globbed and module in universe:
|
||||
globbed.add(module)
|
||||
wake([(module, index, None) for index in range(len(self.parse(module)["items"]))]
|
||||
if self.parse(module) else [], f"{why} glob-imports {name}")
|
||||
visible_to = sees.get(crate, {crate})
|
||||
new = source["paths"] - done["paths"].setdefault(crate, set())
|
||||
done["paths"][crate] |= new
|
||||
for name in new:
|
||||
for target in visible_to:
|
||||
path_names.setdefault(target, set()).add(name)
|
||||
for module in by_module.get((target, name), ()):
|
||||
if module in universe and self.parent[module] in opened:
|
||||
open_module(module, f"{why} names {name}")
|
||||
for key, table, wanted in (("qualified", qualified, lambda other: True),
|
||||
("invokes", invoked, lambda other: item_of(other)["kind"] == "macro"),
|
||||
("mentions", mentioned, lambda other: other[2] is not None or other[0] in globbed)):
|
||||
new = source[key] - done[key].setdefault(crate, set())
|
||||
done[key][crate] |= new
|
||||
for name in new:
|
||||
table.setdefault(name, set()).add(crate)
|
||||
wake([other for other in by_name.get(name, ()) if crate_of(other[0]) in visible_to and wanted(other)],
|
||||
f"{why} {key} {name}")
|
||||
if item["kind"] in TYPE_KINDS and member is None:
|
||||
own = types.setdefault(crate, set())
|
||||
local_types.setdefault(path, set()).update(names(node))
|
||||
if not names(node) <= own:
|
||||
own |= names(node)
|
||||
visible_cache.clear()
|
||||
seeing = seen_by.get(crate, {crate})
|
||||
for name in names(node): # also when the crate has one by that name already: impls of this one
|
||||
wake([other for other in by_type.get(name, ()) if crate_of(other[0]) in seeing],
|
||||
f"implements for {name}, defined in {path}")
|
||||
return files
|
||||
|
||||
|
||||
def impl_subject(header):
|
||||
"""An impl header (the text after `impl`): its generic parameters' names, and the rest without its where clause
|
||||
(the trait and the type it is for)."""
|
||||
header, parameters = header.strip(), set()
|
||||
if header.startswith("<"):
|
||||
depth = 0
|
||||
for index, character in enumerate(header):
|
||||
depth += {"<": 1, ">": -1}.get(character, 0)
|
||||
if depth == 0:
|
||||
generics, header = header[1:index], header[index + 1:]
|
||||
parameters = {match.group(1) for match in re.finditer(rf"(?:^|,)\s*(?:const\s+)?({IDENT})", generics)}
|
||||
break
|
||||
return parameters, re.split(r"\bwhere\b", header)[0]
|
||||
|
||||
|
||||
def crate_of(path):
|
||||
return path.split("/", 2)[1]
|
||||
|
||||
|
||||
def is_workspace_rust(path):
|
||||
parts = PurePosixPath(path).parts
|
||||
return len(parts) > 2 and parts[0] == "crates" and path.endswith(".rs")
|
||||
|
||||
|
||||
DECLARATION_LINE = re.compile(rf"\s*(?:pub(?:\([^)]*\))?\s+)?mod\s+({IDENT})\s*;")
|
||||
|
||||
|
||||
def declarations(text, names, keep_rest):
|
||||
"""A Rust file's `mod` declarations of names, each with the attribute lines just above it; with keep_rest, the
|
||||
whole text except the declarations of other modules instead."""
|
||||
lines, found, attributes = text.splitlines(), [], []
|
||||
for line in lines:
|
||||
if line.strip().startswith("#[") and not line.strip().startswith("#!["):
|
||||
attributes.append(line.strip())
|
||||
continue
|
||||
match = DECLARATION_LINE.match(line)
|
||||
if match and match.group(1) in names:
|
||||
found += attributes + [line.strip()]
|
||||
elif keep_rest and not match:
|
||||
found += attributes + [line]
|
||||
attributes = []
|
||||
return "\n".join(found + attributes)
|
||||
|
||||
|
||||
def declarations_changed(root, path, children, base, whole):
|
||||
"""Whether a changed file differs from the commit base in what a harness depends on: for a module parent, its
|
||||
`mod` declarations of the reached children; for a whole file (whole), everything but its declarations of other
|
||||
modules (adding a module beside reached ones changes nothing). True without a base, or if either side is
|
||||
missing."""
|
||||
if base is None:
|
||||
return True
|
||||
try:
|
||||
head = (Path(root) / path).read_text(encoding="utf-8", errors="replace")
|
||||
before = subprocess.run(["git", "show", f"{base}:{path}"], cwd=root, capture_output=True, text=True, check=True).stdout
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return True
|
||||
return declarations(head, children, whole) != declarations(before, children, whole)
|
||||
|
||||
|
||||
def narrow(dependencies, artifacts, graph, changed=()):
|
||||
"""A harness's dependencies narrowed to what its entry points can reach: {dependency: (whole, children)}, where
|
||||
whole says the file counts for its own content (else only through its `mod` declarations of children, as a
|
||||
module ancestor of a reached file) and children are its reached child modules. Without known entry points (a
|
||||
probe or binary the harness built that isn't found here), every dependency is kept whole."""
|
||||
plain = [dependency.split("#keep:")[0] for dependency in dependencies]
|
||||
crates = {PurePosixPath(path).parts[1] for path in plain if path.startswith("crates/")}
|
||||
rust = {path for path in plain if is_workspace_rust(path)}
|
||||
added = {path for path in changed if is_workspace_rust(path) and PurePosixPath(path).parts[1] in crates
|
||||
and path not in rust and (graph.root / path).is_file()}
|
||||
entries = graph.entries(artifacts, rust)
|
||||
if entries is None:
|
||||
return {dependency: (True, None) for dependency in dependencies}
|
||||
missing = {path for path in rust if graph.parse(path) is None}
|
||||
reached = graph.reachable(entries, rust | added | entries) | missing
|
||||
named, named_in_crate = set(), {}
|
||||
for path in rust | added:
|
||||
parsed = graph.parse(path)
|
||||
file_names = set(parsed["file_names"]) if parsed else set()
|
||||
if path in reached:
|
||||
named |= file_names
|
||||
named_in_crate.setdefault(PurePosixPath(path).parts[1], set()).update(file_names)
|
||||
kept = {}
|
||||
for dependency, path in zip(dependencies, plain):
|
||||
if is_workspace_rust(path):
|
||||
if path in reached:
|
||||
kept[dependency] = (True, set())
|
||||
elif not path.startswith("crates/") or path.endswith(("Cargo.toml", "build.rs")) \
|
||||
or PurePosixPath(path).name in named \
|
||||
or PurePosixPath(path).name not in named_in_crate.get(PurePosixPath(path).parts[1], ()):
|
||||
kept[dependency] = (True, None)
|
||||
for path in sorted(added & reached):
|
||||
kept[path] = (True, set())
|
||||
for path in sorted(reached):
|
||||
child = path
|
||||
while child in graph.parent:
|
||||
parent = graph.parent[child]
|
||||
kept.setdefault(parent, (False, set()))[1].add(graph.module_name[child])
|
||||
child = parent
|
||||
return kept
|
||||
Reference in New Issue
Block a user