pg_lake_copy
pg_lake : Copy to/from data lake files
Overview
| ID | Extension | Package | Version | Category | License | Language |
|---|---|---|---|---|---|---|
| 2567 | pg_lake_copy | pg_lake | 3.7 |
OLAP | Apache-2.0 | C |
| Attribute | Has Binary | Has Library | Need Load | Has DDL | Relocatable | Trusted |
|---|---|---|---|---|---|---|
| --sLd-- | No | Yes | Yes | Yes | no | no |
| Relationships | |
|---|---|
| Schemas | pg_catalog |
| Requires | pg_lake_engine pg_lake_iceberg pg_lake_table |
| Need By | pg_lake |
| See Also | pg_parquet aws_s3 pg_bulkload file_fdw pg_ducklake pg_fact_loader pg_csv omni_csv duckdb_fdw pg_duckdb |
| Siblings | pg_lake pg_extension_base pg_extension_updater pg_map pg_lake_engine pg_lake_iceberg pg_lake_table |
Packages
| Linux / PG | PG19 | PG18 | PG17 | PG16 | PG15 |
|---|---|---|---|---|---|
| el8.x86_64 | N/A | N/A | N/A | N/A | N/A |
| el8.aarch64 | N/A | N/A | N/A | N/A | N/A |
| el9.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| el9.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| el10.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| el10.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| d12.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| d12.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| d13.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| d13.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u22.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u22.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u24.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u24.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u26.x86_64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
| u26.aarch64 | N/A | PIGSTY 3.6.0 | PIGSTY 3.6.0 | PIGSTY 3.6.0 | N/A |
Source
github.com/Snowflake-Labs/pg_lake/tree/main/pg_lake_copy
pg_lake-3.7.0.tar.gz
Install
Make sure PGDG and PIGSTY repo available:
Install this extension with pig:
Config this extension to shared_preload_libraries:
Create this extension with:
Usage
Sources:
pg_lake_copy extends PostgreSQL COPY so queries, heap tables, external lake tables, and Iceberg tables can exchange Parquet, CSV, or newline-delimited JSON files with local paths, HTTP endpoints, and configured object stores. It adds behavior through hooks and has no standalone SQL function API.
The reviewed upstream release is 3.6.0, with SQL extension version 3.6 and PostgreSQL 16–19 support. Existing packaged versions are tracked separately. Keep libraries and the query server from the same release.
Enable the Component
The normal entry point installs pg_lake_copy and its exact dependencies together:
Its control file requires pg_lake_engine, pg_lake_iceberg, and pg_lake_table. The deployment also needs pg_extension_base in shared_preload_libraries and a running pgduck_server.
Export and Import
Format is inferred from the path suffix or selected explicitly:
CSV and compressed output use standard-looking COPY options extended for the lake writer:
The destination can be a PostgreSQL heap table or an Iceberg table; the source can likewise be any query supported by the installed pg_lake stack.
Format and Runtime Boundaries
- Parquet is columnar and preserves supported typed values; CSV and newline-delimited JSON have format-specific inference and conversion options documented upstream.
- Object-store access runs through
pgduck_server. Its credential chain, network access, and bucket permissions must permit the requested read or write. COPYis one statement and participates in the surrounding PostgreSQL transaction, but remote files and cleanup also depend on the pg_lake transaction/queue machinery. Inspect failed operations and orphan cleanup before retrying large exports.- Version
3.5adds no standalone SQL API inpg_lake_copy; the3.4to3.5migration is empty. COPY behavior still depends on matching lake libraries and query-server binaries.
Upstream 3.6.0
Upstream 3.6.0 adds PostgreSQL 19 support and uses DuckDB 1.5.6. Deploy matching libraries and pgduck_server, keep pg_extension_base preloaded, restart loaded processes and apply the supplied SQL updates to version 3.6. This component’s 3.5-to-3.6 SQL migration is empty; the matching runtime and its dependencies still need the coordinated release upgrade.
Upstream 3.6.2
The 3.6.2 patch corrects byte-by-byte CSV-value transmission in pgduck_server. Deploy the matching query-server binary with the extension libraries and reconnect/restart processes holding old code. The component SQL version remains 3.6; an unchanged extversion does not prove the query-server fix is installed. Package availability remains a separate catalog fact.
Upstream 3.7.0
The current reviewed release is 3.7.0 and this component’s control version is 3.7. Earlier version sections describe historical behavior and package examples. Upgrade the complete family with matching libraries and pgduck_server, restart processes holding old code, and apply the supplied extension updates to SQL version 3.7. This does not establish package availability.