cquery最近改动与libclang.so一字节补丁

cquery最近改动

可执行文件从build/app变到build/release/bin/cquery了，支持release/debug/asan等多种waf variants，使用RPATH
./waf configure --bundled=5.0.1可以用上最新出炉的clang+llvm 5.0.1
Riatre把Windows构建修好了 #154
FreeBSD可以使用了 #155及third party库改动，感谢ngkaho1234把sparsepp FreeBSD kvm搞定
不需要在initializationOptions里指定resourceDir了，感谢jiegec的#137
各种模板改进和function template/class template内函数引用的支持。支持了CXCursor_OverloadedDeclRef函数调用#174，但template call template clang-c接口没有暴露相应信息，可能无解。
textDocument/hover信息把函数名插入到函数类型里。用了一些heuristics处理_Atomic decltype() throw() __attribute(()) typeof()，碰到-> int(*)()这种还是没救的，数组括号也不好，但大多数情况都显示得不错的。
加入了实验性的--enable-comments，检索注释。#183 #188 #191 注释和原来的声明信息一起显示，带来了UI的挑战。
VSCode使用浮动窗口，显示多行textDocument/hover不成问题。但Emacs lsp-mode和LanguageClient-neovim就遇到一些困难https://github.com/autozimu/LanguageClient-neovim/issues/224 https://github.com/emacs-lsp/lsp-ui/issues/17
workspace/symbol模糊匹配 #182
danielmartin自己的repo加了实验性的textDocument/formatting，格式化。这必须用clang C++ API，作者有一些顾虑。

用了一个O(n^2) sequence alignment算法，根据编辑距离、camelCase等启发因素排序候选符号(func,type,path,…)。以foo bar为模式会返回fooBar foobar foozbar等，而fooBar排在前面。Emacs xref-find-apropos会自作聪明地把模式用空格分割后当作正规表达式转义，需要覆盖掉。

libclang `handleReference` null pointer dereference

下面介绍重点，libclang一字节补丁。

This is a longstanding issue bothering us, with all the 3 bundled clang+llvm versions: --bundled-clang={4.0.0,5.0.0,5.0.1}.

topisani raised the topic in https://gitter.im/cquery-project/Lobby and I fianlly made up my mind to diagnose it.

https://github.com/jacobdufault/cquery/issues/192

% cd /tmp

% git clone https://github.com/nlohmann/json

% cd json

% (mkdir build; cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..)

% ln -s build/compile_commands.json

% nvim test/src/unit-constructor1.cpp

By appending --log-file /tmp/cq.log to the cquery command line, we can see the error message in /tmp/cq.log:

1 2	indexer.cc:1892 WARN\| Indexing /tmp/json/test/src/unit-iterators2.cpp failed with errno=1 libclang: crash detected during indexing TU

errno=1 indicates CXError_Failure，libclang uses a sigaction+setjmp based crash recovery mechanism to recover from SIGSEGV and returns CXError_Failure. Sadly cquery does not report enough diagnostics for this. For these files, there is no hover, definition, or references information.

https://github.com/jacobdufault/cquery/issues/219 HighCommander4 narrowed it down to a very simple reproduce：

template <typename>
struct actor;
template <template <typename> class Actor = actor>
struct terminal;

default template argument = actor causes the null pointer dereference.

In https://github.com/llvm-mirror/clang/blob/master/tools/libclang/CXIndexDataConsumer.cpp#L203

handleReference(ND, Loc, Cursor,
                dyn_cast_or_null<NamedDecl>(ASTNode.Parent),
                ASTNode.ContainerDC, ASTNode.OrigE, Kind);

dyn_cast_or_null<NamedDecl>(ASTNode.Parent) may return NULL. In some code executed later on https://github.com/llvm-mirror/clang/blob/master/tools/libclang/CXIndexDataConsumer.cpp#L935:

1 2	ContainerInfo Container; getContainerInfo(DC, Container);

getContainerInfo tries to cast DC, which uses a field in DC and causes a null pointer dereference.

I have sent the patch to clang upstream for review, but clang+llvm 5.0.1 was just released. The holiday season is approaching and it is unrealistic to get this submitted and get a new release in the near future. For Linux users who do not want to build clang+llvm from source, my makeshift is ./waf configure --bundled-clang=5.0.1 with

% printf '\x4d' | dd of=build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5.0 obs=1 seek=$[0x47aece] conv=notrunc
% printf '\x4d' | dd of=build/release/lib/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.4.0 obs=1 seek=$[0x4172b5] conv=notrunc

What?

Description

Let’s check the problematic function in libclang:

bool CXIndexDataConsumer::handleReference(const NamedDecl *D, SourceLocation Loc,
                                      CXCursor Cursor,
const NamedDecl *Parent,
const DeclContext *DC,
const Expr *E,
                                      CXIdxEntityRefKind Kind) {
if (!CB.indexEntityReference)  
return false;
if (!D)
return false;
if (Loc.isInvalid())
return false;

According to System V x86-64 ABI, the arguments of this function are passed in the following way:

this: rdi

D: rsi

Loc: rdx

Cursor: stack

Parent: rcx

DC: r8

E: r9

Kind: stack

cquery sets CB.indexEntityReference to OnIndexReference so this field is guaranteed to be non NULL.

Scroll down the assembly listing a little bit and search for the if (!CB.indexEntityReference) condition:

% objdump -M intel -Cd build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5 --start-address 0x47ae90 --stop-address 0x47b190

......

47aeb0: 45 31 ed xor r13d,r13d

47aeb3: 45 85 ff test r15d,r15d

47aeb6: 0f 84 19 03 00 00 je 47b1d5

47aebc: 48 85 ed test rbp,rbp

47aebf: 0f 84 10 03 00 00 je 47b1d5

47aec5: 49 8b 44 24 18 mov rax,QWORD PTR [r12+0x18] # load `IndexerCallbacks &CB`, which is actually a pointer

47aeca: 48 8b 40 38 mov rax,QWORD PTR [rax+0x38] # load CB.indexEntityReference

47aece: 48 85 c0 test rax,rax # a redundant check to see if it is null; we replace it with check of `DC`

47aed1: 0f 84 fe 02 00 00 je 47b1d5 <clang::cxindex::CXIndexDataConsumer::handleReference(clang::NamedDecl const*, clang::SourceLocation, CXCursor, clang::NamedDecl const*, clang::DeclContext const*, clang::Expr const*, CXIdxEntityRefKind)+0x345>

This redundant if statement produces 0x47aeca test rax,rax. If we replace it with if (!DC) (because DC may be NULL and we should avoid null pointer dereference), since DC is passed in the register R8, we may use test r8,r8:

1 2	48 85 c0 test rax,rax 4d 85 c0 test r8,r8

It is now clear that we only need to patch one byte, i.e. the aforementioned printf+dd hack. For radare2 users, r2 -nwqc 'wx 4d @ 0x47aece' build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5.0.

Sadly radare2 dropped the ball when it was in need and I had to resort to printf+dd… Its assembling of test r8,r8 was incorrect https://github.com/radare/radare2/issues/9071😢