Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More work on the graph viewer #1111

Merged
merged 11 commits into from Mar 2, 2022
Merged

More work on the graph viewer #1111

merged 11 commits into from Mar 2, 2022

Conversation

aeisenberg
Copy link
Contributor

@aeisenberg aeisenberg commented Feb 1, 2022

Built on #705.

The viewer is largely implemented now with the following features and
limitations:

  1. Any query with @kind graph will be opened as a graph
  2. Queries that are @kind graph and
    @tags ide-contextual-queries/print-cfg will be used in the
    CodeQL: View CFG context command. This will be visible
    similar to how the AST viewer works. If there is not exactly
    1 such query for a given language, then the extension will throw
    an error.
  3. Unfortunately, the cg viewer assumes that the entire file will
    be added to the graph, so often this will be too big, That leads to
    the following limitation:
  4. There is no size checking on the graph. Graphs that are too big will
    crash vscode.
  5. Lastly, there is a small bug in how the @id is interpreted. Any
    @id with a / in it will place the .dot in a location that
    can't be found by vscode. So, just don't name your queries with any
    /.

This feature is only available in canary mode.

@aeisenberg aeisenberg requested a review from as a code owner Feb 1, 2022
@aeisenberg aeisenberg removed the request for review from Feb 1, 2022
@aeisenberg aeisenberg marked this pull request as draft Feb 1, 2022
@aeisenberg aeisenberg force-pushed the aeisenberg/graph-viewer branch 2 times, most recently from ee498c5 to 1f90e04 Feb 2, 2022
@aeisenberg aeisenberg changed the title [WIP] Graph viewer More work on the graph viewer Feb 2, 2022
@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Feb 2, 2022

@hvitved, would you be able to take a look at this? I think it is mostly what you want (except for the limitations mentioned above).

@hvitved
Copy link
Contributor

@hvitved hvitved commented Feb 2, 2022

I have tested the extension with this query, on a Ruby test DB, and that works just fine, except it keeps printing "Rendering graph" even after it has been rendered:

image

I also tested with this Print CFG query, running the "View CFG" command, but that results in Graph is not available. (an empty graph, I believe).

Thanks again for picking up this task.

@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Feb 2, 2022

Thanks for verifying.

Yes...the rendering graph message is something I need to fix.

And see my point (5) above about why the print ast query is not working. The most correct fix would be from inside of the CLI, which I don't have capacity to do right now. Another work around could be done in vscode itself. I'll see if I can get to that. First priority is getting this feature released internally.

@hvitved
Copy link
Contributor

@hvitved hvitved commented Feb 3, 2022

And see my point (5) above about why the print ast query is not working. The most correct fix would be from inside of the CLI, which I don't have capacity to do right now. Another work around could be done in vscode itself. I'll see if I can get to that. First priority is getting this feature released internally.

I agree that this should not be a blocker. I tried changing the ID to rb-print-cfg, but that still produces an empty graph.

@aeisenberg aeisenberg force-pushed the aeisenberg/graph-viewer branch 2 times, most recently from 6cc7476 to 573fa28 Feb 11, 2022
The viewer is largely implemented now with the following features and
limitations:

1. Any query with `@kind graph` will be opened as a graph
2. Queries that are `@kind graph` and
   `@tags ide-contextual-queries/print-cfg` will be used in the
   `CodeQL: View CFG` context command. This will be visible
   similar to how the AST viewer works. If there is not exactly
   1 such query for a given language, then the extension will throw
   an error.
3. Unfortunately, the cg viewer assumes that the entire file will
   be added to the graph, so often this will be too big, That leads to
   the following limitation:
4. There is no size checking on the graph. Graphs that are too big will
   crash vscode.
5. Lastly, there is a small bug in how the `@id` is interpreted. Any
   `@id` with a `/` in it will place the `.dot` in a location that
   can't be found by vscode. So, just don't name your queries with any
   `/`.

This feature is only available in canary mode.
@aeisenberg aeisenberg force-pushed the aeisenberg/graph-viewer branch from 573fa28 to d1362bf Feb 11, 2022
@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Feb 11, 2022

Using this variant of the print AST query for cpp, I was able to get a graph to be viewed:

/**
 * @name Print AST
 * @description Outputs a representation of a file's Abstract Syntax Tree. This
 *              query is used by the VS Code extension.
 * @id cpp-print-ast
 * @kind graph
 * @tags ide-contextual-queries/print-ast
 */

import cpp
import semmle.code.cpp.PrintAST
import definitions

/**
 * The source file to generate an AST from.
 */
// external string selectedSourceFile();

class Cfg extends PrintASTConfiguration {
  /**
   * Holds if the AST for `func` should be printed.
   * Print All functions from the selected file.
   */
  override predicate shouldPrintFunction(Function func) {
    func.getFile().getBaseName() = "cpu.c" and func.getName() = "sandbox_read_fdt_from_file"
  }
}

I also removed the Rendering graph... text and got the tests passing (I think).

@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Feb 22, 2022

Lat thing I want to do here is just add a message in the graph viewer itself on what the limitations are and where to go for more support so that if external users do start using this, we can set their expectations appropriately.

aeisenberg added 3 commits Feb 25, 2022
Do this by actually walking the interpretation directory.

Move the directory walker from tests to prod and make it async. Also
add tests for it.

And add a warning on graph views to let users know that it is not
production quality.

Finally, change the interpreted directory to be `graphResults` instead
of `interpretedResults.sarif`.
@aeisenberg aeisenberg force-pushed the aeisenberg/graph-viewer branch 2 times, most recently from 684a4b6 to 6c78c2b Feb 27, 2022
@aeisenberg aeisenberg force-pushed the aeisenberg/graph-viewer branch from 6c78c2b to 64b33b7 Feb 27, 2022
@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Feb 28, 2022

OK... finished and ready for a review. This has been fixed:

5. Lastly, there is a small bug in how the @id is interpreted. Any
@id with a / in it will place the .dot in a location that
can't be found by vscode. So, just don't name your queries with any
/.

Also, there is a warning on the results view whenever you view a graph. And I have ensured that this feature is disabled if the canary flag is off.

@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Mar 1, 2022

Here is a sample query that I have been using for the graph:

/**
 * @name Print AST
 * @description Outputs a representation of a file's Abstract Syntax Tree. This
 *              query is used by the VS Code extension.
 * @id id/cpp-print-ast
 * @kind graph
 * @tags ide-contextual-queries/print-ast
 */

import cpp
import semmle.code.cpp.PrintAST
import definitions

class Cfg extends PrintASTConfiguration {
  override predicate shouldPrintFunction(Function func) {
    func.getFile().getBaseName() = "cpu.c" and func.getName() = "sandbox_read_fdt_from_file"
  }
}

It is adapted from the cpp printAST query. Replace the RHS of the getBaseName comparison with a cpp file in a source archive and getName with a name of a function.

If you want to try out the View CFG command, choose any print AST query and add a new tag ide-contextual-queries/print-cfg. Then go to a small source file in a database of the same language, right click and select View CFG.

aeisenberg added 2 commits Mar 1, 2022
Without these changes, a race condition was sometimes hit when viewing
a graph. There are two, related issues that are fixed. These problems
did not appear in the past since rendering a normal results view is
much faster and the message handler is always already set up by the
time the interface first sends a message over to the web view.

1. `vscode.postMessage({ t: 'resultViewLoaded' });` was being called
   before the component is completely mounted. Ie- `componentDidMount`
   is not called. So, the interface is notified that the web view is
   ready to receive messages _before_ it is actually ready to receive
   messages.

   The change ensures the interface only sends messages when the web
   view is ready.

2. `this._panelLoaded` is never set to false if the panel is unloaded.
   This means that if a panel is re-opened, the interface assumes that
   the view is nearly _immediately_ ready to receive messages.

   The change ensures that the interface waits for the webview to really
   be loaded before sending messages.

In both of these cases, if the interface sends the `setState` message
too early, then the message is ignored since no handlers have been added
to the web view.
@aeisenberg aeisenberg removed their assignment Mar 1, 2022
@aeisenberg aeisenberg requested review from cklin and shati-patel Mar 1, 2022
@aeisenberg aeisenberg self-assigned this Mar 1, 2022
@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Mar 1, 2022

@shati-patel, @cklin this is ready for your review now. I fixed a race condition that sometimes prevented a graph from appearing. See the second to last commit.

@aeisenberg
Copy link
Contributor Author

@aeisenberg aeisenberg commented Mar 1, 2022

Forgot to mention that if you want to view a graph, you need to regenerate the web view. To do that, run npm run build in the vscode-codeql/extensions/ql-vscode directory.

cc: @cklin since I'm not sure if you're aware of this.

cklin
cklin approved these changes Mar 1, 2022
Copy link
Contributor

@cklin cklin left a comment

The commits I reviewed are in line with our discussion this morning.

for await (const d of await fs.opendir(dir)) {
const entry = path.join(dir, d.name);
if (seenFiles.has(entry)) {
continue;
Copy link
Contributor

@cklin cklin Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how execution would ever reach this line. Can fs.opendir return duplicate directory entries for the same name?

Copy link
Contributor Author

@aeisenberg aeisenberg Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah...good point. This is a leftover from when I had originally thought it would be handling symlinks. With symlinks, you need to be careful about cycles, but not the case for regular files. I will remove.

The check for `seenFiles` is not necessary since we do not need to
handle symbolic links.
Copy link
Contributor

@shati-patel shati-patel left a comment

Thanks for the PR, and for your helpful walkthrough 🎉 Looks good, and I've had a brief poke around locally too, which works as expected 🙌🏽

@aeisenberg aeisenberg merged commit c18f795 into main Mar 2, 2022
21 checks passed
@aeisenberg aeisenberg deleted the aeisenberg/graph-viewer branch Mar 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants