The Python here is doing exactly what it should - extracting nested data and flattening it into proper relational tables. Most of the 'Python' is actually just the return statement defining table schemas. The schema can be defined separately and reused across API calls. With such reuse the Python part would be really compact. This is the right balance between declarative config and the flexibility needed for real-world API responses. I can confirm it from my experience developing many integrations with Sequor - this approach handles real-world complexity much better than pure declarative tools.

Scroll down and you’ll see the example with raw SQL.

You're right about the abstraction concern. The vast majority of the workflow stays in structured YAML - Python is only needed for two specific points: constructing HTTP request bodies and parsing JSON responses. These are inherently dynamic operations that need real programming logic. The alternatives - proprietary DSLs or visual builders - would be far more complex to learn and maintain than a few lines of Python/JavaScript for JSON manipulation. We actually started with JavaScript since it's more natural for JSON work, but the difference was marginal. Do you have suggestions for a better approach to these two specific dynamic parts? We're open to ideas that would be simpler than Python but still handle the complexity generically.

Fair point - 'intuitive' is subjective and depends on your background. Let me explain where each technology fits: HTTP request configuration (URL, headers, parameters) is pretty intuitive and similar to Postman that most people know. Python is used mainly for response parsing - without a general-purpose language in this place it would be very limiting. Templating is mainly for variable substitution, familiar to anyone using frameworks like dbt. And SQL stays separate from Python - they're not mixed together.

Thank you for pointing this out! I wasn't aware of Arazzo before. It does look very similar indeed. I'll definitely look more into it and see how we might align with their format or adapt it to Sequor's capabilities and execution model. Really appreciate the heads up!

Looks interesting. Has someone done an implementation of this?

Great question! Rate limiting and concurrency are absolutely critical for production API integrations. Here's how Sequor handles these challenges: Rate Limiting:

* Built-in rate limiting controls at the source level (requests per second/minute/hour): each http_request operations refers to an http source defined separately * Automatic backoff and retry logic with delays * There is a option for per-endpoint rate limit configuration since different API calls can have different limits * Because it is at the source level, it works properly even for parallel requests to the same source.

The key idea is that rate limits are handled by the engine - no need to handle it explicitely by the user.

Concurrency is explicit by the user: * Inter-operation parallelism is activated by adding begin_parallel_block and end_parallel_block - between these two operations all the operations are executed at once in parallel * Intra-operation parallelism: many operations have parameters to partition input data and run in parallel on partitions. For example, http_request takes an input table that contains data to be updated via API and you can partition the input table by key columns into specified number of partitions.

Thanks for the Steampipe reference! That's a really interesting approach - exposing APIs as Postgres tables is clever, and I'm definitely going to play with it.

This looks more website oriented, i.e. building HTML pages within a YAML construct. OP's project seems to be more API/middleware oriented.

Totally understand - you want clean, unified data for business insights, not another integration tool to maintain. Sequor actually grew out of our Master Data Management (MDM) work where data cleaning and deduplication are core challenges. We focused on API integration for this release, but have mature data cleaning/deduplication components that we plan to open source as well. What specific data quality issues are you dealing with? Happy to share what we've learned from MDM projects and our data quality engine.

Thank you for letting me know! I'm not seeing any issues on my end - both https://docs.sequor.dev/ and https://sequor.dev/ are loading fine for me. This might be a temporary issue or could be related to your location/network. Try to refresh it. Let me know if you're still having trouble!