As reported in issue #981, using std::regex in IWYU has caused a
tremendous performance regression for large mapping files containing
regex mappings.
$ cat t.cc
#include <string>
# with llvm::Regex
$ time include-what-you-use -Xiwyu --mapping_file=qt5_11.imp t.cc
...
real 0m0,529s
user 0m0,509s
sys 0m0,020s
# with std::regex
$ time include-what-you-use -Xiwyu --mapping_file=qt5_11.imp t.cc
...
real 0m29,870s
user 0m29,717s
sys 0m0,012s
qt5_11.imp contains 2300+ regex mappings, and <string> has a bunch of
includes, so this is a good testbed for regular expression engines, but
over 50x slower is not the result we were hoping for.
The reason we switched to std::regex was to get support for negative
lookaround (llvm::Regex does not have it), but exotic regexes in
mappings are pretty rare, and this is a significant performance hit.
Introduce a --regex option to select regex dialect, with documented
tradeoffs. Put the default back to LLVM's fast implementation.
This fixes issue #981.
If a symbol was ultimately mapped to a header which was (or should be)
included via a ""-style relative include then that would not be handled
correctly.
Update the symbol mappings to be handled in the same way as the include
mappings so that this should no longer be an issue.
When a header was included via a ""-style include relative to the
including header, and that header was marked private, the privateness
wasn't noticed, because the conversion to quoted include was variable
(it could be used as-written, or converted from a path with or without
the context of an including file).
Add a new visibility map for paths rather than quoted includes so these
can be tracked thereby.
It's important that GetCandidateHeadersForFilepath can return the
path itself as a candidate for accessing a specified path.
This used to work by GetPublicValues returning the provided key as one
of the possibilities for accessing that key.
However, the key is not always a quoted include, it can also be a
symbol. So, to prevent GetPublicValues returning a symbol, this
required a workaround where all symbols had to be marked private.
We can simplify all of this by moving the relevant logic out of
GetPublicValues and into GetCandidateHeadersForFilepath. There was
already a case in GetCandidateHeadersForFilepath where the path itself
was added to the candidates, so we simply add an additional circumstance
in which that happens.
This simplifies both GetPublicValues and AddSymbolMapping.
Generalize IncludePicker::MarkVisibility to take a visibility map as an
argument, and rename filepath_visibility_map to include_visibility_map_.
This is in preparation for adding a new visibility map which is keyed by
paths rather than quoted includes.
Fix some corner cases that can arise due to interaction between pragma
export and includes that are relative to the including header (i.e.
includes that can only be done using "", not <>).
Because headers are mostly handled as quoted includes for the purposes
of mapping, this can go wrong with relative includes because they are
not always relative to the same place, and at some places in the code we
don't even know what they're relative to.
Fix this by tracking full paths in parallel or instead of quoted
includes in certain places.
Currently this is just a simple wrapper around a quoted include. This
provides some type safety, but no behavioural change is intended.
This lays the groundwork for adding more data to the MappedInclude
object in order to fix some bugs.
This is part of Google's original open-sourcing of IWYU.
They keep third-party libraries in a directory called, well,
'third-party', so there was lots of special casing for code in
that directory.
Remove that code, unit tests covering it and explicit mappings.
Keep one special case for allowing include cycles for files with
'internal/' in the name, to avoid breaking the include_cycle test
case. Not sure what to do about that longer term, but I didn't want to
remove the test case right now.
This has been a long-standing issue with IWYU (see issues #5, #271 and
probably others) where ConvertToQuotedInclude generates absolute paths
if IWYU is invoked with the current directory different from the source
file path.
This patch moves most path building to use absolute paths and path of
the includer (rather than the cwd) to produce better results.
- Remove DEVTOOLS_MAINTENANCE_ from header guards, that was a
now-unnecessary Googleism
- Fix header guard to match filename in all production code
- Fix header guard to match path in all tests
For some STL containers it is hardcoded how to handle template arguments. But
code detecting specified containers cannot detect these containers in libc++.
This commit fixes detecting containers with precomputed template arguments in
libc++.
--no_default_mappings will cause IWYU not to add the default GCC-oriented mappings, to make it easier to build custom mapping suites for other toolchains.
- Remove hard-coded mapping search path in iwyu_globals.cc
- Remove AddMappingFileSearchPath
- Add private AddMappingsFromFile to handle search path building
- Push search path scanning out into free function FindFileInSearchPath
- Less, but more focused logging.
- Fix indentation in iwyu_include_picker.cc
- Simplify parsing code by making current_node a Node* instead of a Node&
- iwyu.gcc.imp is no longer added automatically if no mapping file is specified
- Instead, the defaults are hard-coded in iwyu_include_picker.cc
- Fix some 80-column overruns
Search for mapping files:
- in the current working directory;
- in the directory where IWYU executable is located;
- in the directory where another discovered mapping file is located.
This was motivated by an attempt to understand how iwyu_ast_util.cc
depended on iwyu_output, which turned out to be the use of VERRS.
I split the VERRS definition out into a separate iwyu_verrs module.
In the fallout from this I discovered some circular dependencies
that this CL attempts to disentangle.
R=csilvers
DELTA=603 (349 added, 222 deleted, 32 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=3892
If no friend is already included when symbols from the file are needed, then iwyu
will just bite the bullet and include the private header.
R=csilvers
DELTA=128 (120 added, 1 deleted, 7 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1983
Also makes it port.h's responsibility to export fnmatch
(before, it was the responsibility of either port.h or
iwyu_include_picker.cc, depending on the platform).
R=dsturtevant,csilvers
DELTA=152 (45 added, 4 deleted, 103 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1852
rather than the one clang gives us. Normally they're the
same, but can be different when the #include could be accessed
via different paths, or via symlinks (for instance, if
we #include "a/b/c.h" and compile with "-I. -Ia -Ia/b", then
we could say #include "a/b/c.h", #include "b/c.h", or #include
"c.h"). clang will, as I understand it, pick one of these
three forms arbitrarily for FileEntry::getName. We store the
name as it was actually typed in the source, and prefer it.
Obviously, the above only works for includes that already
exist. If we suggest a new include, we will fall back on
whatever clang gives us, which is an arbitrary name (in
practice, the first form seen).
R=klimek
DELTA=161 (100 added, 1 deleted, 60 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1851
First, the 'friend' pragma wasn't working at all, due to some
confusion about what needed to be a quoted include and what
needed to be a filepath. I've cleaned all that up, updated
the comments, and added some tests to catch that situation.
Second, I moved the handling of /internal/ to a bespoke scheme
to using the normal 'friend' scheme -- basically, all internal
files automatically are friends with everyone else in the same
'package'. This fixes a bug that caused us to map some
#include to <built-in>, when the include-chain was:
<built-in> -> foo/internal/bar.cc -> foo/internal/baz.h
Before this change, both foo/internal/bar.cc and
foo/internal/baz.h were considered private, since they are in
/internal/, so the #include of baz.h was mapped to the only
non-private "file" in the chain, which is <built-in>. After
this change, foo/internal/bar.cc is still considered private,
but it doesn't matter since it's also considered a friend of
foo/internal/baz.h, and thus allowed to #include it.
R=wan,dsturtevant
DELTA=131 (83 added, 22 deleted, 26 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1728
// IWYU pragma: friend <glob>
Filenames matching the glob are allowed to include the file, which has presumably
been declared private.
R=wan,csilvers
DELTA=123 (106 added, 7 deleted, 10 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1719
map omitting the necessary quotes around the filename. The
check for this was suppressed because we shared the same code
for the filename map and symbol map. Separate out that code
so we can do the check and avoid future problems.
DELTA=30 (6 added, 10 deleted, 14 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1595
treated as private headers, and mapped to the nearest
including public header, was too strict. The basic problem:
what if 'foo/internal/baz.h' tried to include
'foo/internal/bar.h'? It should be able to. In some cases,
'foo/internal/bar.h' isn't even included from any non-internal
file, and we end up suggesting to #include <built-in> (the
only non-private include in the include-chain).
I fixed this up by adding a new function for mapping private
headers to public, that takes into account who is doing the
including. If foo/x/y/z is including foo/internal/a/b/c, we
don't say foo/internal/a/b/c is private in this context. But
if joe/otherproject tries to include foo/internal/a/b/c, then
we *do* say foo/internal/a/b/c is private, and map it to its
closest public header.
I also took out unnecessary code that marked includer-files
that are '/internal/' as private, not just included-files. We
should never need to mark includers as private; if the
includer-file is itself included in turn, we'll have ample
opportunity to mark it private then. Otherwise, we run the
risk of a file being marked private, with nobody including it
that we can map to.
To better match the new semantics that files aren't
intrinsically public or private but it depends on the context,
I renamed GetPublicHeader* to GetCandidateHeader*,
R=wan,dsturtevant
DELTA=179 (84 added, 2 deleted, 93 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1590
likely we'll suggest adding an internal third-party header.
Basically, we stop trying to do include-what-you-use fixes on
third-party code.
More precisely, we make an 'implicit' judgment on which
third-party headers are public and which are private, based on
what existing code (in this translation unit) #includes. We
marked all unincluded files as private, which means iwyu will
never suggest adding a new third-party file as an #include.
Insted, it will suggest some already-included third-party file
that gets the needed file transitively.
Since it's not really practical for us to fix third-party code
to have better #include hygiene, or even to mark up third-party
code with iwyu pragmas, we need to do something similar to
this. We could just manually update iwyu_include_picker's
third_party_map with rules for every third party package we
have, but that's expensive. This is much cheaper, with the
downside that we may miss some potential include-what-you-use
opportunities in third-party code (going from a more-generic
third-party include to a less-generic one). I think that's a
low cost.
R=dsturtevant
DELTA=113 (104 added, 0 deleted, 9 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1577
includes-as-written rather than the actual file-path of the
included file. Since the include-picker deals with actual
file-paths (of the decls), this made no sense, and indeed we
were seeing when code depended on a search path, we weren't
finding the proper include-mapping for it.
For instance, python .h files has
#include "dictobject.h"
rather than
#include "third_party/python2_4_3/gcc-3.4-glibc-2.11.1-grte-k8-linux-python2.6-opt/include/python2.6/dictobject.h
Thus, while we had code that correctly mapped
third_party/python2_4_3 to <Python.h>, it wasn't firing on the
above code because the include-picker saw it as just
"dictobject.h". This is now fixed by using the actual
file-path.
While testing, I discovered the test-file was often calling
AddDirectInclude() improperly (with extra "'s). It didn't
happen to matter, but I cleaned it up.
R=wan,dsturtevant
DELTA=43 (5 added, 2 deleted, 36 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1574
test some main()-invariant logic while I'm at it.
DELTA=19 (16 added, 0 deleted, 3 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1409
thought I was saying the location is where the . (or ->) is,
which is what I want, but clang doesn't actually expose that.
So I have go through some hoops to try to figure it out.
We were actually seeing a problem with this when running:
blaze build --host_cpu=k8 --compile_only -k --crosstool_top=//third_party/llvm/crosstool --plugin=//devtools/maintenance/include_what_you_use:run_iwyu //gws/plugins/local/src:enhanced_listing_ad
It has 'msg_->MSG_foo' in it, where MSG_foo is a macro. We
were attributing this use to the file defining MSG_foo, rather
than to us. With this change, we properly attribute it to us.
R=dsturtevant
DELTA=150 (125 added, 6 deleted, 19 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=1347
public/private information in a separate data structure from
the mapping. Besides being a bit clearer to follow, this
makes it easier to verify that we're consistent in declaring a
file public or private. We now do verify this, which turned
up a few inconsistencies in the hard-coded data (not anything
major), which I've fixed.
This prompted a bit of a change in the API, to separate out
adding of mappings from adding of public/private-ness. This
fits the iwyu preprocessor workflow better, allowing us to
resolve a TODO or two.
I've also cleaned up a comment or two.
R=wan
DELTA=186 (81 added, 60 deleted, 45 changed)
Revision created by MOE tool push_codebase.
MOE_MIGRATION=722
* Add support for FakeNamedDecls for tests (dsturtevant)
* Write our own symbol sanitizer rather than getQualified... (csilvers)
* Change run_iwyu_tests to better fit python test framework (csilvers)
* Revamp IWYU pragmas, now support keep + export + private (dsturtevant)
* Fix a bug when nested classes are defined out of line (csilvers)
* Add the ability to print an arbitrary ASTNode (csilvers)
All code by csilvers was reviewed by wan and dsturtevant. Code by
dsturtevant was reviewed by csilvers.
1) Removed the use of <tr1/tuple> -- it wasn't getting us that much,
and cost us in portability. Resolves
http://code.google.com/p/include-what-you-use/issues/detail?id=2
2) Removed unused 'using' statements, renamed some vars to be cleaner,
added 'const' in a few places I could, etc.
3) Refactored some logic into a new function GetCalleeFunctionType().
4) Made a few tests more robust by using better fakes (mostly in
more_tests, so not directly relevant to the opensource code).
5) Added a check to make sure we don't add any new include-mappings
after we were supposed to have finialized the mappings.
All code reviewed by wan and/or dstur...
Kim Gräsman