---
title: "Sources, metadata, and reproducible exports"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Sources, metadata, and reproducible exports}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
library(mongolstats)
knitr::opts_chunk$set(eval = FALSE, collapse = TRUE, comment = "#>")
```

This guide shows the 0.4 workflow for keeping official statistics connected to
their source, selected dimensions, observation symbols, and collection context.
All service requests below are examples that you run deliberately; building
this vignette does not access the network.

## Choose the source

| Service | Use | Identifier |
|---|---|---|
| PXWeb, `data.1212.mn` | Published statistical tables | `DT_NSO_...` |
| Public data portal, `data.nso.mn` | Experimental aggregate series and datamart discovery | `portal:DT_NSO_...` in discovery results |
| Metadata catalog, `metadata.nso.mn` | Definitions, forms, classifications and relationships | A typed numeric ID |

A shared title or table-like identifier does not establish that two services
publish the same measure. Geographic codes also belong to their source. For
example, the sampled portal GDP series uses national code `CO9999`; the reviewed
PXWeb resident-population template uses national code `0`.

## Start with a reviewed concept

`nso_indicators()` searches a small bundled registry of reviewed PXWeb concepts.
Inspect the definition, unit status, validation date, and caveats before fetching.
The query template fixes the substantive selections; period labels are supplied
separately. It does not guess that a similarly named portal series is equivalent.

```{r curated-concepts}
old_options <- nso_options(mongolstats.lang = "en")
nso_indicators("GDP")
concept <- nso_indicator("gdp_current")
concept[c("name", "definition", "unit", "unit_status", "verified_at", "caveats")]

query <- nso_indicator_query("gdp_current", periods = c("2023", "2024"))
gdp <- nso_fetch(query, labels = "both")

# Both sexes and national total are the reviewed defaults.
population_query <- nso_indicator_query("resident_population", periods = "2024")
options(old_options)
```

Use period labels such as `"2024"`, rather than PXWeb positional time codes:
code `"0"` can refer to a different year after the next update. A source label
ending in `*` is preserved. Read the source's explanation of that mark before
assigning a revision status or converting the published number's scale.

## Inspect source information

Fetch results remain ordinary tibbles. Their provenance attribute records source
identity, request URL, selections, retrieval time, available source metadata,
and a metadata fingerprint. `nso_table_info()` provides a compact view;
`nso_provenance()` exposes the full manifest, including failures in a batch.

```{r inspect-provenance}
nso_table_info(gdp)
manifest <- nso_provenance(gdp)
manifest$records[[1]]$selections
manifest$failures
nso_citation(gdp)

# Metadata-only lookup: no observations are requested.
nso_table_info("DT_NSO_0500_001V1")
```

An absent source statement, update date, or unit remains missing. A metadata-only
lookup can contain less information than a data response. Citations acknowledge
the source; the package's software license does not determine the data's reuse
conditions. Consult [NSO's terms](https://www.nso.mn/en/terms_of_use) and the
relevant source record.

## Keep source symbols

Numeric `NA` alone cannot distinguish a missing observation from source symbols
such as `..`. `nso_value_status()` returns observation keys, the original
`value_raw`, and `value_status`: `"observed"` for finite numbers, `"missing"` for
empty or missing fields, and `"symbol"` for other text. A symbol's meaning is
not inferred. Retain these records before filtering or transforming data.

```{r observation-integrity}
status <- nso_value_status(gdp)
status[status$value_status != "observed", , drop = FALSE]

nso_check_data(
  gdp,
  keys = "Year",
  period = "Year",
  expected_periods = c("2023", "2024")
)
```

This example uses English PXWeb dimension names; inspect `names(gdp)` when using
another fetch language. Optional `unit` and `geography_level` arguments name
columns already present in your data. The checker reports concerns without
changing values. No reported concerns does not establish that populations,
units, or geographic levels are comparable.

## Export a durable record

Transformations and CSV exports can discard attributes. Save the provenance
sidecar and original status table separately. The JSON manifest contains source
and request context, not the statistical observations or their raw symbols.

```{r save-sidecars}
# Choose an enduring project directory; use a new run directory for each export.
export_dir <- file.path("outputs", "nso", format(Sys.Date(), "%Y-%m-%d"))
dir.create(export_dir, recursive = TRUE, showWarnings = FALSE)
# Do not overwrite a previous run accidentally.
targets <- file.path(export_dir, c("gdp.csv", "gdp.rds", "value-status.rds", "provenance.json", "analysis-context.rds"))
stopifnot(!any(file.exists(targets)))
saveRDS(gdp, file.path(export_dir, "gdp.rds"))
analysis_context <- list(
  saved_at = format(Sys.time(), tz = "UTC", usetz = TRUE),
  data_sha256 = digest::digest(gdp, algo = "sha256"),
  session = utils::sessionInfo(),
  code_revision = Sys.getenv("ANALYSIS_CODE_REVISION", unset = NA_character_),
  manual_recodes = character()
)
saveRDS(analysis_context, file.path(export_dir, "analysis-context.rds"))
utils::write.csv(gdp, file.path(export_dir, "gdp.csv"), row.names = FALSE)
saveRDS(nso_value_status(gdp), file.path(export_dir, "value-status.rds"))
nso_write_manifest(gdp, file.path(export_dir, "provenance.json"))

saved_manifest <- nso_read_manifest(file.path(export_dir, "provenance.json"))
nso_table_info(saved_manifest)
nso_citation(saved_manifest)
readRDS(file.path(export_dir, "value-status.rds"))
```

`nso_write_manifest()` refuses to overwrite a file unless `overwrite = TRUE` is
explicit. The RDS file preserves the original R types and attributes. Record the analysis code revision and any manual recodes explicitly. A metadata fingerprint alone cannot reproduce observations after source revisions.

## Apply checks before calculating rates

The other tutorials load the inspectable teaching helpers shipped at
`system.file("examples", "analysis-helpers.R", package = "mongolstats")`.
They check expected unit/month grids, unique keys, common periods, complete age
strata and positive denominators before calculating estimates. Missing source
observations remain unknown. `nso_check_data()` reports concerns; an empty report
does not prove that an analytical specification has been met.

Keep the original fetched object, its `nso_value_status()` result and provenance
before transformations. Save the validated analytical data and any recoding rules
alongside the outputs above. Historical teaching snapshots without retrieval dates
or raw symbols cannot acquire that missing information retrospectively.

## Find definitions and collection context

The metadata catalog connects databases, tables, questionnaire fields, forms,
and classifications. Its `indicator` type can be a question in a collection
form rather than a published series. Numeric catalog IDs do not identify PXWeb
tables. Search coverage and pagination attributes describe what was returned,
not a verified complete inventory.

```{r metadata-discovery}
hits <- nso_metadata_search("хүн ам", type = "all")
attr(hits, "coverage")
attr(hits, "pagination")

organizations <- nso_metadata_search(type = "organization")
databases <- nso_metadata_search(organization = 1, page = 1)
attr(databases, "pagination")

database <- nso_metadata("database", 526, include_raw = TRUE)
database$methodology
database$access
nso_related(database)

field <- nso_metadata("indicator", 12925, include_raw = TRUE)
field$quality_flags
field$context$unit
field$context$value_type
field$date_fields
nso_related(field)
```

The numeric IDs here are concrete examples from inspected catalog records;
choose records relevant to your analysis. `nso_related()` reads embedded
relationships without further requests. Fetch a full supported record with
`nso_metadata(type, id)`; embedded details may be partial. Quality flags preserve
suspect source fields for review rather than silently correcting them.

```{r forms-and-classifications}
forms <- nso_forms(database_id = 526, include_raw = TRUE)
forms$records
form <- nso_metadata("form", 1024, include_raw = TRUE)
form$dates
form$years
form$date_fields
form$context
form$quality_flags

classification <- nso_classification(3446, include_raw = TRUE)
attr(classification, "completeness")
attr(classification, "quality_flags")
attr(classification, "metadata")$version

nso_metadata_dictionary("frequency")
nso_metadata_dictionary("collection-method")
nso_metadata_dictionary("licence")
```

Classification codes remain character strings. An empty codebook does not prove
that the classification has no categories, and a nonempty one does not prove
completeness. A licence or security category describes the catalog record;
it does not grant access to underlying records. Missing official translations
remain missing.

## Check discovery freshness

Local PXWeb discovery uses the bundled or session index. Its entry count can
include the same table under several paths. Live keyword search returns a
source-defined search window; it does not rebuild the complete index.

```{r catalogue-freshness}
nso_catalogue_status()
nso_search("population", fixed = TRUE)
live_hits <- nso_search("population", live = TRUE)
attr(live_hits, "catalogue")

config <- nso_api_config(refresh = TRUE)
config[c("maxCells", "maxValues", "origin", "status")]
```

The package splits data queries by the cell limit and applies conservative
request pacing. `maxValues` describes the metadata-value limit advertised by
PXWeb; it is not a data-query selection limit. Configuration can fall back to
defaults or a stale cache, so inspect `status`. Processes on one computer share
request pacing; other computers using the same public address need coordination.

## Use the public portal explicitly

The portal registry is experimental. Inspect its `queryable`, `quality_flags`,
and grouping dimensions; rows with invalid or missing IDs remain discoverable
but cannot be queried. Portal request parameters are separate from the curated
PXWeb templates, and the source's summary narratives are not observations.

```{r portal-series}
portal_catalog <- nso_indicators(source = "portal", refresh = TRUE)
portal_catalog[c("id", "name", "unit", "queryable", "quality_flags")]
nso_indicator("portal:DT_NSO_0000_4_4")

portal_gdp <- nso_portal_data(
  "DT_NSO_0000_4_4",
  area = "1",
  dimension = "code",
  graph_type = "line",
  periods = "2024",
  labels = "both"
)
portal_gdp[c("source_id", "period", "code", "code_mn", "code_en", "value", "unit")]
nso_check_data(portal_gdp, keys = c("period", "code", "code1"), unit = "unit")
nso_table_info(portal_gdp)
```

`area`, `dimension`, and `graph_type` are explicit source request parameters;
the GDP example does not establish their meaning for every indicator. Period
filtering is performed after retrieval and matches the source label exactly.
Codes are retained for every label choice. Missing English labels stay `NA`;
unknown geography codes are preserved. Unpadded periods such as `2025-1` retain
their label but remain undated when month-versus-quarter meaning is ambiguous.

```{r datamart-discovery}
datamarts <- nso_portal_datamarts()
datamarts[c("id", "name", "years", "source", "query_supported")]
```

Datamart support provides catalog discovery and presentation configuration only.
The reported year range is source text, not verified coverage. `query_supported`
is `FALSE`: a stable developer contract for aggregate queries has not been
verified, and this interface does not retrieve underlying individual records.