Thanks for the insightful comments! You surely have a lot more experience than me there, but my impression was that producing visual diffs and merging files are tasks that put different requirements on the tree matching algorithms, and Dijkstra-style approaches felt more fitting for diffs than for merging, so that's why I went for GumTree as it seemed to be the state of the art for merging. Does SemanticDiff offer a merge driver? I could only find documentation about diffing on the website.

As to mismatches: yes, they are bound to happen in some cases. Even for line-based diffing, Git uses rather convoluted heuristics to avoid them (with the "histogram" diff algorithm), but they can't be completely ruled out there either. I hope that with enough safeguards (helper to review merges, downstream consistency checks with local fall-back to line-based diffing) they can be lived with. I'm happy to try other matching algorithms if they are more promising though (there isn't much coupling with the rest of the pipeline).

Concerning tree-sitter, I have noticed some small issues, but nothing that was a show-stopper so far. I actually like it that it's designed for syntax highlighting, because it's really helpful that the representations it gives stay faithful to the original source, to avoid introducing reformatting noise in the merging process. Parsers written for a specific language can sometimes be too zealous (stripping comments out, doing some normalizations behind your back). That's a problem in Spork (which uses Spoon, a pretty advanced Java parser). And the uniform API tree-sitter offers over all those parsers is just too good to give up, in my opinion.

> We are currently in the process of moving away from tree-sitter and instead using the parsers provided by the languages themselves where possible.

I imagine this means you're trying to abstract over those parsers somehow? How well is that going, and have you written about your approach?

(I wrote `resholve` to identify and rewrite references to external dependencies in bash/posixy Shell scripts to absolute paths. This is helpful in the Nix ecosystem to confirm the dependencies are known, specified, present, don't shift when run from a service with a different PATH, etc.

It builds on the mostly-bash-compatible OSH parser from the oilshell/oils-for-unix project for the same reasons you're citing.

It would be ~nice to eventually generalize out something that can handle scripts for other shell languages like fish, zsh, nushell, elvish, the ysh part of the oils-for-unix project, etc., but I suspect that'll be a diminishing-return sort of slog and haven't had any lightbulb-moments to make it feel tractable yet.

We also have some ~related needs here around identifying hardcoded or user-controlled exec...)

An important point here is that for certain languages, using the original grammar is pretty much impossible. In particular, for C, you want to do diffing and merging on the un-preprocessed source, but the language's grammar very much assumes the source has gone through the preprocessor.

Of course, the existence of the preprocessor means there are situations where it's completely impossible to know what the correct parse is; it will necessarily be heuristic in some cases.

Hi! ast-grep[1] author here. It is a tree-sitter based syntax tool to search tool.

I wonder how you transition from tree-sitter to other builtin parsers? Tree-sitter gave a unified interface to all languages. Using language native parsers will require significant work for various FFI if I am not wrong.

[1]: https://ast-grep.github.io/

> best effort approach to parsing. This is perfectly fine for syntax highlighting, since the worst that can happen is that a few characters are highlighted incorrectly. However, when diffing or modifying code you really want the code to be parsed according to the upstream grammar, not something that mostly resembles it.

But surely you need to support code that doesn't parse correctly by the actual language's grammar anyway? 'Merge branch fix-syntax-error'

Small brained primate comment.

I've wondered if you could add annotation keywords to languages to convert them into something that could be parsed reliably with a tree sitter grammar.

I say this as someone that feels like you really want diffs that say, changed 'struct x name from y to z' instead of here's a huge list of files with ten changes each.

This may or may not be on your radar, but crypto is desperate for a product like this. Smart contracts are often forks or rewrites (obfuscated or otherwise) of others and an easy interface for end users to be able to see changes between two forks would probably provide a lot of value.

Interesting. My students used Language Server Protocol data to make syntax-aware diffs. Very promising project. Unfortunately, everyone moved on. I am currently looking for ways to revitalize it.

https://github.com/shishyando/tokenized-myers-diff

Good example of when adding abstraction is more problematic than the processes themselves which is like an extra minute or two in flow.

That is nicely done, often hard to find, and usually it's what I'm looking for when deciding whether to use a piece of software: Show me the complexity you've encapsulated so that I can be the judge of whether the juice is worth the squeeze.

Armed with that, I can tolerate some rough edges. Without it, I'll get stuck in weird ways that your docs can't anticipate.

Not only that, the order of fields in a Java class does matter despite what that link claims. It's common to use Lombok to automatically generate constructors, and "the order of the parameters match the order in which the fields appear in your class."

https://projectlombok.org/features/constructor

The first two kinds of conflicts that Mergiraf handles looks somewhat dangerous to me when handled by a computer.

https://mergiraf.org/conflicts.html

Yes, that's definitely something that could be refined, for instance by specifying that only children of specific types can be reordered together: https://codeberg.org/mergiraf/mergiraf/issues/6

I sort my CSS by property type (display, size, position, etc). Takes a while to get used to it but it definitely speeds up my ability to do CSS surgery.

I am very excited about this tool despite agreeing with what you wrote here.

The reason for that is that most of the time when I'm resolving huge numbers merge conflicts... I don't give a shit about details like field order. I just want to get some code that's functionally correct at p<0.05 so I can figure out what performance characteristics my old feature branch would have if I resurrected it. Or I want to kick off the slow integration tests ASAP. 9/10 times it's that kind of thing.

The 1/10 times where I'm like "OK now I actually wanna merge this code, I have to look over the resolutions, I will upload them to Gerrit/GitHub and do a self-review" I am more than happy to spend 20 minutes correcting order etc. Or I'll happily just switch this tool off and do a totally manual merge.

So yeah I think it just comes down to usecase.

I want to add another problem here. <https://mergiraf.org/adding-a-language.html#add-commutative-...>:

> To let Mergiraf reorder using statements to fix conflicts, we actually need to specify that they can be reordered with any of their siblings (any other child of their parent in the syntax tree).

That’s too coarse-grained. No idea about C♯, but languages could impose a rule that imports must come before anything else—long ago Rust had such a rule, for example. So it might be that within your compilation_unit node, only its using_directive children are commutative, and only among themselves.

Otherwise (lapsing into Rust syntax for convenience), with Base `use A; struct X…`, Left `use A; use B; struct X…` and Right `use A; struct Y; struct X…`, you could end up with the invalid `use A; struct Y; use B; struct X…`.

Yes maybe, but these issues are true for Git's native merge algorithms too. It isn't perfect either.

As soon as you do any merge you're accepting that there might be edits you don't agree with.

Could some of this be mitigated by running a prettifier post-merge?

But surely Mergiraf has some opinion about the order when it doesn’t matter, right? Like structs before impls, in your example.

Fyi, giraffes are wild animals, and can be very violent and territorial.

Boa constrictor could be a less friendly mascot.

FWIW your examples are very unclear as they use text formatting, you need to indent lines by 4 spaces (with an empty line before and after) for a code block e.g.

if a == b: print("x")

versus

    if a == b:
        print("x")

I tried your example but git does create a conflict in my case - but maybe I misunderstood the scenario. Python support can likely be done (I would be thrilled if someone made a PR for it), but I don't know if there is a lot of potential for solving conflicts there: imports can have side effects, function arguments are complicated with the mixture of positional and keyword arguments, decorators are effectful… it seems to me that there is a lot of sensitivity to order in many places.

Yeah at the moment it just supports whatever the tree-sitter parser accepts, period. A bring-your-own-grammar version could be interesting, I don't see why it couldn't work. Do you have any Rust crates to recommend, to do parsing according to a grammar supplied by the user at run time? It's likely to be slower, but maybe not prohibitively so…

Another approach would be for the tool to accept doing structured merging even if there are error nodes in the parsed tree. If those error span the parts of the file where the extended language is used, then the tool could still help with merging the other parts, treating the errors as atomic blocks. I'd be a bit reluctant to do that, because there could be errors for all sorts of other reasons.

This sounds like what jujutsu's workflow is like by default

git absorb is great, use it all the time.

Have you tried p4merge? It's usually one of the first tools I install.

Yes, see this example: https://mergiraf.org/conflicts.html#line-based-merges

Out of curiosity, which languages would you be interested in?

difftastic does only diffing not merging.

It's based on tree-sitter indeed: https://mergiraf.org/adding-a-language.html

what do you mean?

its treesitter based already:

https://mergiraf.org/architecture.html

No typescript either, but it does support JS.

There are attempts to do that, such as https://arxiv.org/abs/2111.11904 or https://arxiv.org/abs/2109.00084. I couldn't find any open source implementation though. I'm personally not so keen to use LLMs for merging: I want it to be quick and predictable.