René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Defer slow parts of scanning","articleBody":"The scan phase should be fast and seldom fail because it is not parallelized and does not have separate logs for each package. It is also possible for Java source units detected by the scanner to be ignored (e.g., using `SkipDirs` in the Srcfile); in that case it is undesirable to incur the wait to download all of an ignored module's deps.\n\n(In this case, \"fast\" means \"takes hundreds of milliseconds and doesn't hit the network.\")\n\nAlso, I can't find anywhere that I wrote this down, but I initially had in mind that scanning would always be fast and some other design decisions in srclib assume that it is fast. For example, to make srclib usable for realtime editing, it needs to be able to run the scan phase whenever the project's configuration changes, which could be often (e.g., a file is added or deleted). If scan is slow, then we'd either have to wait a long time to re-scan each time the project configuration changes, or we would have to move more scan-related logic into the grapher and other tools (e.g., the grapher might assume responsibility for knowing that `src/**/*.java` are the Java files). The former is obviously not good and the latter means that the grapher now has to be entangled with the build system (a big goal of the separation was to avoid having to write a Maven grapher, a Gradle grapher, etc.).\n\nCurrently the srclib-java toolchain fetches all of the Maven/Gradle deps in the scan phase. We should defer that work to the graph and depresolve phases. (There needs to be some way to share work between those 2, since the graph phase almost always needs to resolve deps.)\n","author":{"url":"https://github.com/sqs","@type":"Person","name":"sqs"},"datePublished":"2014-12-20T10:28:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/27/srclib-java/issues/27"}