r-data-io

Use arrow::read_parquet() to read and arrow::write_parquet() to write parquet files in R. For efficiency, use col_select to read only needed columns, or arrow::open_dataset() with dplyr filters for row-level pushdown.

Parquet is the recommended default for tabular data because it is columnar, compressed, type-preserving, and readable by Python, Spark, and DuckDB. Use CSV only for small human-readable data and Excel for business stakeholder exchange.

Use pool::dbPool() with an odbc driver to create a connection pool with configurable minSize and maxSize. The pool automatically manages connection checkout and return, and you close it with pool::poolClose() when finished.

Yes, DuckDB can query parquet files directly using read_parquet() inside SQL statements through DBI. This lets you run aggregations and filters on disk-resident parquet files without loading full datasets into R memory.

Hardcoded absolute paths break when code runs on different machines or CI environments. Use here::here() or fs::path() to build portable, project-relative paths that work regardless of where the project is cloned.

Store credentials as environment variables in a gitignored .Renviron file and read them with Sys.getenv() in code. For CI and production, use GitHub secrets or vault integration instead of committing credentials to the repository.