Found while measuring healthbR against microdatasus and read.dbc for the
comparison vignette (SIH-SP 2023-01, 210,225 rows; same results before and
after, compared with identical()):
parse = TRUE converts each distinct value of a column once and spreads
the result back, instead of running as.Date() on every row. DATASUS
date columns repeat a few hundred values over hundreds of thousands of
rows; date parsing was 2.6 s of a 5 s warm sih_data() call. Applies to
every DATASUS module.sih_data(2023, 1, uf = "SP") went from 7.7 s to
2.3-2.8 s in the same session.sih_data(lazy = TRUE) opens only the requested years of the mirror
instead of listing the whole SIH prefix (11,184 objects, 12.7 s before
any data was read), and lists the newest of them once instead of twice.
Deaths per competence for a year of Roraima went from 16.7 s to 3.7 s in
a new R session; a national two-year query from ~20 s to ~12 s.vignette("healthbr-vs-microdatasus") (same address) now compares
healthbR, microdatasus and read.dbc: what each layer does, read.dbc's
removal from CRAN (2025-12-14) and how to install it, speed measured on
the same SIH files (cold, warm and decompression only), the same tasks
written in each, and the healthbr-data Parquet mirror read directly with
arrow or duckdb.sipni_data(uf = "RJ", source = "datasus") for a 2020+ month used to hold
all 27 UFs of the national monthly CSV in memory until the end of the read,
so campaign months (the June 2021 file has 41.8M rows, 21 GB of CSV)
needed more than 16 GB even for one state (#4, reported by
@EnfRodrigoBruno). With arrow installed the file is now re-encoded to
UTF-8 byte-wise into ~256 MB pieces (arrow's dataset scanner reads UTF-8
only, and handed the whole file it reads ahead of the writers without
bound) and each piece is streamed by arrow into the partitioned cache,
one partition per UF, and the
requested state is read back from there: no row of the national file is
materialised in R, whatever its size. The cache still ends up with all 27
UFs of the month (a later request for another state does not download the
~1.4 GB ZIP again), and the partitions are written to a staging directory
first, so an interrupted read never leaves a partial UF in the cache.
Without arrow a readr chunked reader keeps only the requested state and
discards the others (the partitioned CSV cache was never readable without
arrow anyway). Both engines read every column as character and treat
"" and "NA" as missing, so the cache holds the same values whichever
filled it.
Found while measuring #4 against the real June 2021 file: the national CSV
now carries the 56-field schema of the JSON exports, whose UF column is
sg_uf_estabelecimento. The reader only knew sigla_uf_estabelecimento
and uf_estabelecimento, so every row went to a pseudo-UF "ALL" and the
call returned an empty tibble after reading the whole file (had it not run
out of memory first). The reader now recognises the three headers, matches
the UF case-insensitively, and when none is present says so from the
header alone, before reading the file.
https://data.sidneybissoli.com first (the bucket behind Cloudflare's
cache, same paths) and fall back to the r2.dev URL when the domain does
not answer. options(healthbR.r2_pub_base = ...) overrides the domain.
sipni_info() points to the domain.sih_years() now starts in 1992 instead of 2008. The healthbr-data mirror
has carried the whole SIH-RD series (1992-present) since it was built, but
sih_data() refused any year before 2008 even with source = "r2",
because the year list was written for the DATASUS FTP folder
(SIHSUS/200801_). The FTP fallback still covers 2008 on only and says so
when asked for an earlier year. Two caveats for the early years, both
properties of the source files: RACA_COR (race/colour) only exists from
2008 on, and DIAG_PRINC is CID-10 only from 1998 (1992-1997 use the
6-digit CID-9 derived coding). Needed by the sih-br-mcp cubes for
1998-2007.First CRAN release since 0.2.0: this version also carries everything in 0.3.0 and 0.3.1 below (SI-PNI on the healthbr-data mirror, SI-PNI dictionary corrections), which were GitHub-only.
vignette("healthbr-vs-microdatasus") compares healthbR with
microdatasus on the
DATASUS systems both read (SIM, SINASC, SIH, SIA, SINAN, CNES): coverage
table, what each package does with the raw codes (labels vs. codes plus
dictionary), where the bytes come from (DATASUS FTP .dbc vs. the
healthbr-data Parquet mirror with provenance), lazy evaluation, cache, and
a measured run of the same three downloads (SIH RR 2023-01, SIM and SINASC
AC 2022) in both packages. README and DESCRIPTION now name DATASUS and the
neighbouring packages (microdatasus, PySUS, basedosdados) so the
comparison is findable.sih_status() tibble cost ~4.6 s and ran on every sih_data(source = "r2") call, so a one-file read from a warm cache took ~5.5 s against
0.9 s from the FTP. The per-partition output-file lookup is now done once
and the summary is memoised per manifest content (rlang::hash()), the
way the parsed manifest already was: the same warm read now takes ~0.8 s
and sih_status() ~0.75 s. Found while measuring the comparison vignette.The healthbr-data mirror
(hive-partitioned Parquet on Cloudflare R2, one partition per DATASUS file
RD{UF}{yy}{mm}.dbc, values byte-identical, 1992-present, provenance record
per file) is now the default source of the SIH module, the same design the
SI-PNI module adopted in 0.3.0:
source argument in sih_data(): c("r2", "datasus") (default)
reads from the mirror and falls back to the DATASUS FTP when the mirror
yields nothing; a single value pins a source. r2_credentials points at
another bucket. Without \pkg{arrow} the FTP is used directly. Both
transports share the local partitioned cache.sih_status() (new): one row per published partition (competence x
UF) from the mirror's manifest, with the DATASUS URL, MD5 and size of the
source file, record count and processing timestamp -- the way to see
which competences exist and whether a file was re-issued. It also carries
the Parquet behind each partition (parquet_path, parquet_sha256,
parquet_size_bytes) and the healthbr-data pipeline that wrote it
(pipeline_version, git_commit), with the manifest's last_updated
and manifest_version as attributes -- enough for a derived product to
record exactly which files it was built from (the sih-br-mcp cubes do).attr(x, "healthbr_source") says which source
served the tibble; from the mirror, attr(x, "healthbr_provenance") lists
the source files behind it.sih_data(lazy = TRUE) now returns a remote
arrow (or duckdb) dataset over R2 -- nothing is transferred until
collect(), and filters and column selections are pushed down. The
mirror's 14 historical schemas are handled by imposing the schema of the
newest requested year, so only the requested years are ever read.year/month are the AIH's billing
competence, not the admission date (DT_INTER), and that the four
competences after a year close 99.7-99.9% of its admissions (measured on
the whole mirror).*_clear_cache() now clears the partitioned cache too. The shared
.clear_cache() removed only flat <module>_*.parquet|rds files and left
the hive-partitioned dataset directories (sih_data/, sim_data/,
sipni_r2_data/, ...) behind, so sih_clear_cache() answered "No cached
SIH files to clear" while sih_cache_status() still listed the partitions
-- and a "cold" download timed right after clearing was served from the
cache. It now removes the same directories .cache_status() reports and
says how many partition files went. Affects every module with a
partitioned cache. Found while measuring the comparison vignette.PRELIM/ to
FINAIS/; the package still built PRELIM/ URLs for 2023-2024, which no
longer exist on the FTP, so sinan_data(2023)/sinan_data(2024) failed.
Year lists updated: final = 2007-2025, preliminary = 2026.Detected by the portfolio source monitor (edition probing against the
DATASUS FTP); the year lists in R/*_data_internal.R remain the source of
truth until discovery is made dynamic.
09 is Haemophilus
influenzae tipo b (Hib) per the official .cnv, not BCG (BCG is data
code 02) — and used DOSE/FX_ETARIA codes that do not occur in the data
at all. Any decoding done with the built-in dictionary of previous
versions should be redone. Regeneration script: data-raw/sipni-dictionaries.R.lookup = TRUE in sipni_dictionary(): returns a data-code
lookup (one row per value as it appears in the data, expanding the .cnv
source_codes — including comma lists like 88,45 and ranges like
11-31), ready to join against sipni_data() results. Joining data
against the .cnv code column decodes to wrong labels; the vignette
examples were corrected accordingly. Overlapping categories are resolved
by specificity — explicitly listed codes beat range catch-alls — so
residuals like FX_ETARIA "Idade ignorada" (source codes 00-99) only
label codes no specific category claimed.unify_schemas = TRUE
(all modules): when cached years carry different column sets — DPNI 1994
has 7 columns, 2019 has 12 — reads used the first file's schema and
silently dropped the extra columns of other years.year, month, uf_source) first, matching fresh downloads.sipni_cache_status() (and every module's *_cache_status()) now also
reports hive-partitioned cache datasets, which were previously invisible.\u escapes into Rd files
(R CMD check WARNING).pnadc_apply_survey_design() error tests skip without
srvyr.The old OpenDataSUS host (arquivosdadosabertos.saude.gov.br) was
decommissioned and, as of 2026, the Ministry removed the 2020-2025 microdata
files from the new location as well — sipni_data() for years >= 2020 had
been failing with 404 (#1). This release makes the
healthbr-data mirror
(hive-partitioned Parquet on Cloudflare R2, values byte-identical to the
Ministry's files, complete 2020+ series) the default source for the whole
SI-PNI module:
source argument in sipni_data() and sipni_dictionary():
c("r2", "datasus") (default) reads from the R2 mirror and falls back to
the official DATASUS/OpenDataSUS sources automatically; pass a single
value to pin a source. r2_credentials allows pointing at another
S3-compatible bucket.sipni_status() (new): month/UF-level availability and provenance of
the mirror, read from its manifest.json files (processing timestamp,
Ministry source URL, record counts). Manifests are cached locally and
revalidated by ETag.healthbr_source (which source
served each era) and, for R2 reads, healthbr_provenance (per-partition
processing timestamp + source URL).dt_vacina, ds_vacina, ...), while
the OpenDataSUS CSVs use different names (data_vacina,
descricao_vacina, ...). Each source returns its columns exactly as
published — no renaming. See sipni_variables(type = "API", source =).lazy = TRUE with R2 returns the remote arrow dataset itself: dplyr
verbs are pushed down and only the touched partitions are transferred
(partition columns keep the bucket layout: ano, mes, uf).source = "datasus"); the R2 dictionaries are the full versions
converted from the Ministry's original .cnv/.dbf files, with a
source_codes traceability column.arrow package (in Suggests); without it,
sipni_data() falls back to DATASUS with a message.CNES (National Health Facility Registry): Health facility data from DATASUS FTP as .dbc files (2005-2024). 13 file types (ST, LT, PF, DC, EQ, SR, HB, EP, RC, IN, EE, EF, GM), monthly data per UF.
SINAN (Notifiable Diseases Surveillance): Disease notification microdata from DATASUS FTP as .dbc files (2007-2024). 31 diseases including DENG, CHIK, ZIKA, TUBE, HANS, HEPA, MALA, and more.
SI-PNI (National Immunization Program): Aggregated vaccination data from DATASUS FTP as .DBF files (1994-2019), plus individual-level microdata from OpenDataSUS CSV (2020-2025). Transparent year-based routing between sources.
SISAB (Primary Care Health Information): Coverage indicators from the relatorioaps REST API. 4 report types (aps, sb, acs, pns) at 4 geographic levels (brazil, region, uf, municipality).
ANS (Supplementary Health): Beneficiary, complaint, and financial data
from the ANS open data portal. Plus ans_operators() for operator registry.
ANVISA (Health Surveillance): Product registrations, pharmacovigilance, hemovigilance, technovigilance, and controlled substance sales (SNGPC) from the ANVISA open data portal. 14 data types total.
POF (Household Budget Survey): Food security, consumption, and health expenditure microdata from IBGE FTP (2002-2003, 2008-2009, 2017-2018).
PNADC (PNAD Continua): Health-related supplementary modules from PNAD Continua (deficiencia, habitacao, moradores, APS).
PNS (National Health Survey): Microdata and SIDRA tabulated data (2013, 2019).
Censo Demografico: Population denominators from the IBGE SIDRA API, covering Census years 1970-2022 and intercensitary estimates 2001-2021.
Hive-style partitioned parquet caching for all DATASUS modules.
Lazy evaluation with lazy = TRUE and backend = "arrow" or "duckdb"
for out-of-memory queries.
Parallel downloads via .map_parallel() (furrr when configured, purrr
fallback).
Download progress bars for all modules using cli::cli_progress_bar().
Shows spinner, current/total count, progress bar, and estimated time
remaining (ETA) during multi-file downloads.
Concurrent HTTP downloads for SI-PNI multi-month CSV requests via
curl::multi_download().
Resumable HTTP downloads via Range headers for large files.
Smart type parsing (parse = TRUE) for all 7 DATASUS modules, with
automatic numeric and date column conversion. Use col_types for
fine-grained control or parse = FALSE for all-character.
Extracted shared helpers for validation, search, cache management, and return logic, eliminating ~1100 lines of duplication across modules.
Reduced cyclomatic complexity across 11 high-complexity functions by extracting internal subfunctions. All exported and internal functions now have cyclomatic complexity below 15.
Removed deprecated flat cache migration infrastructure.
Moved readxl and haven from Imports to Suggests.
Removed unused vroom from Suggests.
Added "Parallel downloads" section to all 12 *_data() help pages
(?sim_data, ?sih_data, etc.) explaining how to enable parallel
processing via future::plan().
Added "Parallel downloads" section to README with a working example.
Added CONTRIBUTING.md with contributor guidelines.
Added CODE_OF_CONDUCT.md (Contributor Covenant v2.1).
Complete refactoring of VIGITEL functions due to Ministry of Health website restructuring. Data is now distributed as a single consolidated file containing all years (2006-2024) instead of separate files per year.
vigitel_data() API changed: new format parameter, year defaults to
NULL (all years), removed lazy/parallel/force_download parameters.
Removed vigitel_download() and vigitel_convert_to_parquet() functions.
pns_sidra_data() unknown table warning.file.exists() guard before file.size() check).Moved arrow package from Imports to Suggests for better cross-platform
compatibility. The package now checks for arrow availability and provides
informative error messages with installation instructions when needed.
Added cache_dir parameter to all data fetching functions (vigitel_data(),
vigitel_download(), vigitel_variables(), vigitel_dictionary(),
vigitel_cache_status(), vigitel_clear_cache()). This allows using
tempdir() for temporary storage that doesn't persist after the R session.
Updated examples to use cache_dir = tempdir() to avoid leaving files on the
system during CRAN checks.
vigitel_years() - list available VIGITEL survey yearsvigitel_variables() - list variables available in a specific yearvigitel_dictionary() - get the data dictionary with variable descriptionsvigitel_data() - download and load VIGITEL data with multiple options:
furrrvigitel_convert_to_parquet() - convert cached Excel files to Parquet formatfurrr package